From b08f372022c50ac34ad1801cee5648468015e254 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 12 Oct 2026 16:38:18 +0200 Subject: [PATCH] docs: retire the SZZ-pool metric, add the valid-benchmark brief, plain-English pass The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in the wrong place about half the time, and ~76% of entries are library-internal (not reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise. Retired as a metric and a gate across the roadmap and briefs; no pool-scored number is quoted anywhere. Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified; ordinary negatives from the same repos; unseen repos; the whole tool measured; plain metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch). Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two active briefs via Codex, then reviewed the diff: only these files changed, every number/path/hash/command is unchanged, and the retired-metric decision is intact. Co-Authored-By: Claude Opus 4.8 --- .../engine-change-scoped-analysis/brief.md | 425 +++++++++--------- .../specs/valid-detection-benchmark/brief.md | 85 ++++ .kiro/steering/roadmap.md | 245 +++++----- README.md | 71 ++- 4 files changed, 455 insertions(+), 371 deletions(-) create mode 100644 .kiro/specs/valid-detection-benchmark/brief.md diff --git a/.kiro/specs/engine-change-scoped-analysis/brief.md b/.kiro/specs/engine-change-scoped-analysis/brief.md index fbd3f73..5d96756 100644 --- a/.kiro/specs/engine-change-scoped-analysis/brief.md +++ b/.kiro/specs/engine-change-scoped-analysis/brief.md @@ -1,93 +1,93 @@ # Engine change-scoped analysis (G2 workstream) -Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks a concrete lever for the -engine-detection gap found while driving the G0 baseline. This is G2 ("improve what moves the unseen-repo -numbers"), gated on the pool metric; it is NOT part of the G0 record-07 baseline (that runs on the quick-rule tier). +Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks work to address the +engine-detection gap found during the G0 baseline run. This is G2 ("improve what moves the unseen-repo +numbers"), gated on the pool metric; it is not part of the G0 record-07 baseline (that runs on the quick-rule tier). ## Problem (measured, not assumed) -On real unseen vulnerability changes the engine produces **no finding** — but the diagnosis (live, 300 s long-deadline -replays) shows it is usually **not a detection-capability gap**: +The engine produces **no finding** on real unseen vulnerability changes. Live 300 s long-deadline replays show +that the cause is usually not a detection-capability gap: - **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`, - `change_context_incomplete`. **Zero analyses ran.** And the ranked candidates were **repo-wide and unrelated to + `change_context_incomplete`. No analyses ran. The ranked candidates were **repo-wide and unrelated to the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to re-audit the whole repo and runs out of budget before analyzing the change. - **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but `comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` + - `contributor-scan ... projection_unavailable`. Multi-causal; correspondence was one part (fixed separately, - commit c39fbb8), capability registry is Task 8.3. + `contributor-scan ... projection_unavailable`. This has several causes. Correspondence was one (fixed separately, + commit c39fbb8); the capability registry is Task 8.3. ## Hypothesis -A pre-push hook should vet **the code this push changed**, not re-scan the whole repository. Seeding/scoping the -engine's analysis to the **changed declarations and their dataflow neighborhood** (the `change_sites` are already in -the replay spec) should let analysis **actually run within budget** instead of being deferred — which is where -detection is being lost, not in Joern's capability. +A pre-push hook should check **the code this push changed**. Starting the engine's analysis from the +**changed declarations and their dataflow neighborhood** should let it run within budget instead of being +deferred. The `change_sites` are already in the replay spec. The missed detection comes from deferral, +rather than a limit in Joern's capability. ## Anchors (confirm in increment 1) -- `src/openultrasast/rank.py` — the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`). -- `src/openultrasast/mapping.py` — builds the `selected`/`deferred` scope + the budget allocation that defers on +- `src/openultrasast/rank.py` is the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`). +- `src/openultrasast/mapping.py` builds the `selected`/`deferred` scope + the budget allocation that defers on `deadline_exhausted`. -- `src/openultrasast/push/policy.py`, `push/runner.py` — consume the scope; change_sites arrive via the push input. -- `benchmarks/unseen/replay_entry.py` — carries `changed_paths` / `declaration_paths` in the spec. +- `src/openultrasast/push/policy.py` and `push/runner.py` consume the scope; change_sites arrive via the push input. +- `benchmarks/unseen/replay_entry.py` carries `changed_paths` / `declaration_paths` in the spec. ## Increments (progress tracking) - [x] **0. Cost split confirmed (2026-10-11, timed live replay).** `build_seconds=66.69` (CPG build is cheap), `query_seconds=0.0` (no query ever executed), total 303 s (hit the deadline), **4,095** candidate questions all `analysis_not_started=deadline_exhausted`, `selected: []`, `regions_scanned=0`. So of the 303 s, ~67 s built the - CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change — - the budget was spent before a single query ran. The CPG build is NOT the bottleneck; the over-broad candidate set - is. Change-scoping is confirmed as the right lever, and because per-candidate prep includes the line-correspondence - work, fewer candidates also cuts that cost (the two fixes compound). + CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change. + The budget was spent before a single query ran. The over-broad candidate set is the bottleneck, rather than the CPG build. + This confirms change-scoping as the fix. Preparing each candidate includes line-correspondence work, + so fewer candidates also reduce that cost. - [x] **1. Located (2026-10-11).** The over-enumeration is upstream of the scan, not inside it: - - `push/runner.py:345-348` — `head_regions = discovery(tip.root, …)` discovers regions across the **whole head + - `push/runner.py:345-348`, `head_regions = discovery(tip.root, …)` discovers regions across the **whole head tree** (the ~4,095), independent of the change. - - `push/runner.py:397-407` — those regions are passed wholesale to `scan_repository(..., change_context=context)`; + - `push/runner.py:397-407`, those regions are passed wholesale to `scan_repository(..., change_context=context)`; `change_context` is threaded in but **never scopes the region set**. - - `model/scan.py:167` and `:364` — `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all + - `model/scan.py:167` and `:364`, `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all regions × families; with `order_by_evidence=True` (`runner.py:288/403`) the scan evidence-tiers **every** pair - before analysis — that pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`. + before analysis. That pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`. - Primitive already present: `model/regions.py:279` computes the changed-path set from the context; `compare_targeted_base` (`runner.py:411`) is a precedent for change-targeted work. - - **Fix locus (increment 2):** restrict the *analyzed* region set to the change's declarations + dataflow + - For increment 2, restrict the *analyzed* region set to the change's declarations + dataflow neighbourhood (filter `head_regions` by `context.changed_paths` before the scan, or scope the enumeration inside `scan_repository`). Keep the **CPG whole-repo** so taint queries can still trace sources from unchanged code into - the changed regions — only the question set is change-scoped. + the changed regions. Only the question set is change-scoped. - **Correction (2026-10-11), after a reverted first attempt — two things the naive filter got wrong:** + Correction (2026-10-11), after reverting the first attempt: the filter got two things wrong. - **The engine already reaches into unchanged code by design, and it is tested.** A changed `source` that reaches an unchanged `sink` must still select the unchanged sink's question (`tests/test_model_change_context.py::test_changed_dependency_reaches_unchanged_ranked_question`; - `test_change_does_not_replace_ranker_selection` pins that change context must NOT alter the ranker's selection). - `_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths` — which locations to itemise vs - summarise — NOT to shrink the tier population. A "changed files only" scope deletes the reach-into-unchanged + `test_change_does_not_replace_ranker_selection` pins that change context must not alter the ranker's selection). + `_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths`. This controls which locations to itemise vs + summarise, rather than shrinking the tier population. A "changed files only" scope deletes the reach-into-unchanged behaviour. Any scope must keep changed regions **plus the regions they reach** (the affected/`relationships` neighbourhood), not changed files alone. - **The 236 s is now split (2026-10-11, saved artifacts).** `total=291s build=52s query=0s arbitrate=238.75s`, - and `query_seconds_by_kind["evidence"]=238.54s` — the evidence-tier pass is **99.9 %** of the arbitrate phase; + and `query_seconds_by_kind["evidence"]=238.54s`, the evidence-tier pass is **99.9 %** of the arbitrate phase; `affected_context` + `_scope_work` are **0.21 s**. The pass tiered 1,015 pairs (`tier_counts 0:1009, 1:6`) at - **~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget — `selected=0`, + **~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget, `selected=0`, `deferred=1365` all `deadline_exhausted`. Secondary anomaly: the design assumes ~2.5 ms/pair - ("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is pathologically slow on a cold large - CPG — a separate optimisation from cutting the pair count. - - **So increment 2's single target is the evidence pass** (`scan.py:338-359`, `_evidence_pass`). The fork is HOW - to cut it without deleting reach-into-unchanged: - - (i) **Scope + bound:** order the evidence candidates change-first and bound the pass to a fraction of the + ("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is unusually slow on a cold large + CPG. Fixing that is separate from cutting the pair count. + - Increment 2 targets the evidence pass (`scan.py:338-359`, `_evidence_pass`). The options must + reduce its cost while preserving flows into unchanged code: + - (i) Scope + bound: order the evidence candidates change-first and bound the pass to a fraction of the budget so analysis queries still run; reached unchanged regions are tiered only while budget lasts. - - (ii) **Neighbourhood pass:** tier the changed regions, read the reached regions from the context rows the - pass already returns, tier those in a second small pass — exact reach-into-unchanged, more code. - - (iii) **Per-pair speedup:** attack the ~235 ms/pair (warm server / batching) rather than the pair count — - helps every scan but is a deeper engine change and does not by itself bound a huge repo. + - (ii) Neighbourhood pass: tier the changed regions, read the reached regions from the context rows the + pass already returns, then tier those in a second small pass. This preserves exact reach-into-unchanged behaviour but needs more code. + - (iii) Per-pair speedup: reduce the ~235 ms/pair (warm server / batching) rather than the pair count. + This helps every scan but needs a deeper engine change and does not by itself bound work on a huge repo. - **CORRECTION (2026-10-11, after implementing (i) + four live re-verifies — the cost model was wrong).** + **CORRECTION (2026-10-11, after implementing (i) + four live re-verifies, the cost model was wrong).** Scope + bound was built and tested (change-first ordering, adaptive budget-fitted portioning, `_context_pass` scoped to changed files, honest `evidence_pass_bounded` degradation; branch `engine-change-scoped-evidence-pass`). - It correctly cut the tiered pair count — but `query_seconds` stayed **0** every time. The live numbers show why: + It cut the tiered pair count, but `query_seconds` stayed **0** every time. The live numbers show why: | pairs tiered | run_batch calls | evidence s | build s | |---|---|---|---| @@ -99,53 +99,53 @@ detection is being lost, not in Joern's capability. Evidence time is **flat regardless of pair count** → the cost is **per-*call* fixed** (~150–220 s: loading the CPG into the query engine + the census the evidenceOnly batch computes), ~85 ms/pair on top. Pair-count scoping therefore cannot free budget, and portioning re-pays the fixed cost per call (so (i) can be counterproductive). - Build is also large and swings 52→127 s — two engines thrashing the 7 GB host (the record-07 baseline shares it). - The real levers are now (iii)-shaped, not (i): **warm/persistent query engine or a cached census** to kill the - ~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s; plus a **clean (uncontended) - measurement**, because contention alone may be inflating build + fixed cost 2–3×. Scope+bound is NOT merged. + Build time also varies from 52→127 s. Two engines compete for the 7 GB host, which the record-07 baseline shares. + The fixes now resemble (iii), rather than (i): a **warm/persistent query engine or a cached census** to remove the + ~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s. A **clean (uncontended) + measurement** is also needed because contention may inflate build + fixed cost 2–3×. Scope+bound is not merged. **UNCONTENDED measurement (2026-10-11, baseline paused via SIGSTOP + container kill, then resumed).** Shipped engine, one evidence call over 1,015 pairs: `build=48 s`, `evidence=243.79 s`, `query=0`. The evidence pass is - **the same ~244 s uncontended as ~238 s contended** → the cost is **INTRINSIC, not contention**; contention only + **the same ~244 s uncontended as ~238 s contended** → the cost is **intrinsic, not contention**; contention only inflated `build` (48 → 127 s) and added run-to-run noise. The model is `evidence ≈ 150 s fixed (CPG load into the query engine + the census the evidenceOnly batch computes) + ~93 ms/pair` (244 = 150 + 1015×0.093). Consequences: - **~200 s of fixed overhead** (`build` + evidence-load) precedes any analysis, so a 300 s budget leaves only ~100 s for everything else. This is why `query=0` even on a clean host. - **Change-scoping still helps, but only by cutting the per-pair tail** (~1,015×93 ms ≈ 95 s): one evidence call - over ~30 changed pairs is ~150 s + 3 s instead of 244 s, which would free ~90 s — possibly enough for analysis, - but marginal. + over ~30 changed pairs is ~150 s + 3 s instead of 244 s. This would free ~90 s, which might leave enough time + for analysis, but only just. - **It must be ONE call, not portioned.** The ~150 s is a first-call load, so the adaptive probe attributed it to - its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4 — the implemented - scope+bound is doubly wrong for this cost model. The correct shape: change-first candidates, count-capped, ONE + its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4, the implemented + scope+bound has both problems under this cost model. The correct shape: change-first candidates, count-capped, ONE evidence call, `deadline=None`. - - **Pre-push viability:** the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a + - Pre-push viability: the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a repo this size without a warm/persistent session or a cached census. A larger budget (e.g. 600 s) makes the benchmark arm run but is not a realistic pre-push deadline. The lever that matters for the product is killing the ~150 s fixed cost, not scoping pairs. - [ ] **2. Seed analysis from the change.** Prioritize (or restrict first-pass to) the changed declarations plus their call/data neighborhood over repo-wide static rank, within budget. -- [ ] **3. Soundness.** Keep cross-file dataflow INTO the changed function (don't scope to literal changed lines); +- [ ] **3. Soundness.** Keep cross-file dataflow into the changed function (don't scope to literal changed lines); add tests; keep existing push tests green. -- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now RUN and produce a finding on +- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now run and produce a finding on `install_packages` (vs `deferred`)? Re-run metrics-plugin. -- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate) — needs the +- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate), needs the kube lane (blocked on the AX completion path) for scale. ## Dependencies / notes -- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; the win - here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding. -- Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine - changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here. +- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; this work + moves findings from `comparison_unknown`/deferred to a decided novelty → advisory engine finding. +- Implement directly and verify on the live instrument (a fake-client/unit pass is not evidence the live engine + changed); the kube-lane and store endpoints are infrastructure owned by the operator. ## Efficiency barrier analysis (post-record-07 priority) -**What the record-07 baseline actually measured (verified on the records): regex only.** The default and -long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0` — **no LLM is -called** — and the engine never completed (`engine_covered=0`, the ~200 s wall). So the measured `catch≈2.5%` is -the **quick pattern-rule tier alone**: essentially grep-with-structure. This is the FLOOR, not the product, and it -confirms the maintainer's point — pattern rules alone will never be enough, or `grep` would do. The product's -detection lives in two tiers that did NOT run: +The records confirm that the record-07 baseline measured regex only. The default and +long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0`. No LLM is +called, and the engine never completed (`engine_covered=0`, the ~200 s wall). The measured `catch≈2.5%` covers +only the quick pattern-rule tier, a structured pattern search. This measures the floor, rather than the product. +It confirms the maintainer's point that pattern rules alone are insufficient; otherwise `grep` would do. +The product's two detection tiers did not run: | Tier | Mechanism | Measured cost (uncontended, mid-size repo) | Ran? | Catch | |---|---|---|---|---| @@ -153,20 +153,20 @@ detection lives in two tiers that did NOT run: | 2 engine (Joern) | CPG build + taint/dataflow | build 48 s + evidence (150 s fixed + 93 ms/pair) + analysis (unmeasured) | no (walled) | unknown | | 3 LLM adjudication | model judgment / search-with-proof (hunter, fusion, decision engine) | model latency + $/call (unmeasured in the push path) | no (disabled) | unknown | -**The barrier:** a releasable net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window — -target **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis is a budget ledger per tier against -that window, and the measurements each needs: +To be releasable, the net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window. +The target is **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis records each tier's budget against +that window and identifies the measurements needed: - **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer. Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session - with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the decisive architectural + with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the architectural test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size. Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail. - **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call - (DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall LIFT of tier 3 over tiers - 1+2 on the pool — does the LLM catch what regex and dataflow miss, and at what $ and latency per push? -- **Deliverable:** a **barrier table** — for budget ∈ {60, 120, 300, 600} s × each tier combination, does it FIT - and what is the MEASURED recall/false-alarm on the pool — identifying the minimal viable config for a releasable + (DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall lift of tier 3 over tiers + 1+2 on the pool, does the LLM catch what regex and dataflow miss, and at what $ and latency per push? +- Produce a **barrier table** for budget ∈ {60, 120, 300, 600} s × each tier combination. Record whether it fits + and its measured recall/false-alarm on the pool. Use this to identify the minimal viable config for a releasable net. 600 s unblocks the benchmark arm but is not a pre-push deadline; the product target is tiers 2+3 under ~120 s. Sequence: finish G0 (record 07) → **E1/E2 first** (they decide whether the warm session alone crosses the engine @@ -176,8 +176,8 @@ lever. ## E1/E2 RESULTS (measured 2026-10-12, CLEAN host, bedrock-agentcore, 3-file change, 300 s budget, head side) -Four clean-host replays settle the engine barrier and **correct the earlier "~200 s graph-load wall"**, which was -conflated with host contention from the running baseline replay: +Four clean-host replays **correct the earlier "~200 s graph-load wall"** diagnosis. It had confused graph-load +cost with host contention from the running baseline replay: | Run | build | evidence (taint batch) | dominance | query_s | selected | deferred | |---|---|---|---|---|---|---| @@ -186,40 +186,40 @@ conflated with host contention from the running baseline replay: | warm session `SESSION=1` (clean) | 32.9 | 228.6 | 25.9 | 25.9 | **356** | 1009 | | session + graph preload (clean) | 30.4 | 223.2 | 32.5 | 32.5 | **356** | 1009 | -**Findings (each traced to a measured cause, not assumed):** +The measurements show: -1. **The engine arm was walled by CONTENTION, not an intrinsic floor.** The prior `query=0 / selected=0 / all +1. Host contention prevented the engine arm from running. The prior `query=0 / selected=0 / all 1365 deferred` was measured while the slice-1 baseline replay contended for the 7 GB host. On a clean host the - **shipped (disposable) engine already runs dataflow**: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**, + shipped (disposable) engine already runs dataflow: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**, dominance flow queries complete. This is the first time the engine arm produced analysis on this change. -2. **The evidence/taint batch (~225 s) is INVARIANT to session and to graph preload** (225.1 / 228.6 / 223.2 across +2. **The evidence/taint batch (~225 s) is invariant to session and to graph preload** (225.1 / 228.6 / 223.2 across the three clean runs). Warm session and preload-at-build do **not** reduce it. So the dominant cost is intrinsic to the taint batch, not graph I/O. -3. **Graph-load is cheap — it is NOT the wall.** The `dominance` query is a full cold query *including* +3. Graph-load is cheap and does not account for the delay. The `dominance` query is a full cold query *including* `importCpg`, and it costs only ~35 s. So `importCpg` is tens of seconds, not ~150 s. **This corrects the earlier note** that attributed the ~150 s fixed to "first query JVM-start + graph-load + census." The ~160 s fixed lives inside `taint.sc`'s own per-batch flow work (seed/sink memo fill over the distinct `(family, file)` combinations - it walks), paid lazily as the first pairs process — independent of the raw pair count. + it walks). It is paid lazily as the first pairs are processed, independent of the raw pair count. 4. **The warm session is not the lever.** It starts fine on a clean host (the contended `session_startup_timeout` was contention-only), but gives ~no benefit here: subsequent queries (dominance) are already cheap disposable, and the evidence pass is a single batch call, so there is nothing for the session to amortise. -5. **The barrier, restated:** `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer - as `deadline_exhausted`; only 356 are analysed. The lever is **reducing the taint batch's work** — change-scope - the pair/family set so the batch fits budget — NOT session, preload, or a bigger JVM. The parked branch +5. The barrier, restated: `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer + as `deadline_exhausted`; only 356 are analysed. The fix is to **reduce the taint batch's work** by scoping + the pair/family set to the change so the batch fits the budget. Session reuse, preload, or a bigger JVM will not do this. The parked branch (`engine-change-scoped-evidence-pass`) is therefore **relevant after all, on a clean host**, provided it scopes the family/file set `taint.sc` walks (not merely the pair list); its adaptive portioning still needs replacing with a fixed cap (per-call-fixed cost, not per-pair). -6. **Side note:** the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run — a separate +6. Side note: the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run, a separate intermittent base-revision build issue; the `head` side built and analysed normally. Track separately. -**What E1/E2 did NOT answer (the real G2 question):** on a clean host the engine selects 356 regions — *does it -CATCH vulns regex misses?* That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the -engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the decisive +E1/E2 did not answer the G2 question: does the engine catch vulnerabilities regex misses when it selects 356 regions +on a clean host? That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the +engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the next experiment, ahead of E3/E4/L1/L2. ## ENGINE-ON PILOT RESULTS (measured 2026-10-12, CLEAN host, 5 public pool repos, vuln change each, 300 s) -Five public repos picked for flow-shaped families and small size (engine's best chance to finish and catch): +Five public repos were picked for flow-shaped families and small size, giving the engine its best chance to finish and catch: | repo | fam | size | REGEX on vuln | ENGINE finding | cov | vuln region ranked | best taint | selected/deferred | wall | |---|---|---|---|---|---|---|---|---|---| @@ -229,70 +229,70 @@ Five public repos picked for flow-shaped families and small size (engine's best | aws/bedrock-agentcore | injection | 3f | ✗ | **none** | incomplete | yes | — | 356 / 1009 | 305 s | | miguelgrinberg/python-socketio | deser | 10f | **✓ (n=2)** | **none** | incomplete | yes | **tier1 score2.0 sink found** | 136 / **0** | 257 s | -**Result: regex 1/5, engine 0/5.** On a clean host, with budget to spare on most, the shipped engine **added no +Result: regex 1/5, engine 0/5. On a clean host, with budget to spare on most, the shipped engine **added no catches over the regex floor.** -**Why (measured, traced — CORRECTED after inspecting `head.findings`, not ranker evidence):** -1. **The budget hypothesis is REFUTED.** banks, graphiti and socketio **finished their region queue** (deferred=0) +The diagnosis was corrected after inspecting `head.findings` rather than ranker evidence: +1. The budget hypothesis is refuted. banks, graphiti and socketio **finished their region queue** (deferred=0) and still emitted no finding. Change-scoping to fit budget would not have changed the outcome. -2. **The engine produced ZERO vuln-family findings (path/injection/deser) on all 5.** Actual `head.findings`: +2. The engine produced no vuln-family findings (path/injection/deser) on any of the 5. Actual `head.findings`: banks 1 (config_secrets), socketio 2 (config_secrets), launch-editor / graphiti / bedrock **0 findings each**. The dataflow/taint tier traced **no source→sink flow** for the vuln families (`taint_evidence_not_applicable`; - e.g. banks found the `path.open` sink but **no source reaches it**). **Correction:** the tier=1 / score=2.0 - "sink" seen in the ranker's `normalized_evidence` is the RANKER's static sink candidate, NOT a confirmed taint - finding — an earlier draft of this section misread it as detection partially working. It was not. -3. **`ambiguous_line_correspondence` is a downstream RED HERRING for these vulns.** It fires on all 5, but there + e.g. banks found the `path.open` sink but **no source reaches it**). Correction: the tier=1 / score=2.0 + "sink" seen in the ranker's `normalized_evidence` is the ranker's static sink candidate, not a confirmed taint + finding. An earlier draft of this section misread it as detection partially working. It was not. +3. `ambiguous_line_correspondence` does not explain these missed vulnerabilities. It fires on all 5, but there are **no vuln-family findings to anchor**, so a region-level witness fallback (the proposed "Option A") would change nothing here. Witness/anchoring only touched the config_secrets findings that did appear. -4. **The LLM adjudication tier was DISABLED** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The +4. **The LLM adjudication tier was disabled** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The rungs reached: `suspicion=0, model_corroborated=0, model_entailed=0–2, execution_confirmed=0`. The tier designed to confirm a vuln the static dataflow cannot trace **never ran**. 5. **Ranking also under-promotes the vuln region** (tier 0 for most families; banks path family stuck low). So even where a sink exists, the region is not analysed deeply enough to attempt a flow. -6. **Measurement caveat:** the pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe +6. The pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe uses tight vuln-introducing diffs. Secondary to (2)/(4). -**Reprioritised engine levers (CORRECTED — anchoring is NOT the lever):** -- **#1 the LLM adjudication tier (disabled in the push path) — now the primary lever.** regex+dataflow = 1/5 with - the model OFF; the static tiers produce no vuln-family findings on these shapes. The product thesis - ([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot is the empirical case to - turn it on: **L1** model latency + calls/push, **L2** recall LIFT on the same 5 (and beyond). Costs money — stage - it (5–10 sizing → ~10–20 pilot → full only on lift). +Reprioritised engine levers (corrected, anchoring is not the lever): +- #1: the LLM adjudication tier (disabled in the push path) has the highest priority. regex+dataflow = 1/5 with + the model OFF; the static tiers produce no vuln-family findings on these shapes. The hypothesis + ([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot supports + testing it: **L1** model latency + calls/push, **L2** recall lift on the same 5 (and beyond). These calls cost money, so stage + them (5–10 sizing → ~10–20 pilot → full only on lift). - **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0). -- **#3 change-anchoring robustness** — real, but only matters AFTER the static/LLM tiers produce vuln-family +- #3 change-anchoring robustness matters only after the static/LLM tiers produce vuln-family findings to anchor. Deprioritised until (1)/(2) yield findings. -**Release read:** the pre-push net is NOT releasable on detection. Clean host, generous budget, LLM off: regex 1/5, -engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the decisive untested lever -is the **LLM adjudication tier**. Next: an **LLM-tier pilot (L1/L2)** on these 5 repos — does enabling the model -produce vuln-family findings regex+dataflow miss, and at what $/latency. +Detection does not meet the pre-push net's release requirements. Clean host, generous budget, LLM off: regex 1/5, +engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the untested lever +is the **LLM adjudication tier**. Next, run an **LLM-tier pilot (L1/L2)** on these 5 repos to test whether the model +produces vuln-family findings regex+dataflow miss, and measure its $/latency. ## ROOT CAUSE of engine 0/5 + the "#2 guided by #1" workstream (2026-10-12) -**Why the taint tier traced no flow (verified in code + evidence):** the pilot repos are **libraries**, whose +Code and evidence show why the taint tier traced no flow. The pilot repos are **libraries**, whose untrusted input arrives as **public-API function parameters**, not HTTP requests. The Python taint **source** vocabulary (`ruleset/semantic/python.toml`) is entirely web/CLI: Flask/Django `request.*`, `sys.argv`, `sys.stdin`, `input(`. There is no source for a library's own API parameter. The engine *does* have a parameter-as-source mode (`_parameters_are_input`, `scan.py`), but it fires **only for recognised entry points** (`_is_entry_point` and -`entry_kind != "function"`) — the pilot vuln regions carry `entry:false`, so their parameters are **deliberately +`entry_kind != "function"`). The pilot vuln regions carry `entry:false`, so their parameters are **deliberately not** sources. That restriction is intentional: treating every function parameter as a source caused a flood of -false SQL-injections on WordPress/PMPro (documented in the `_parameters_are_input` docstring). So **the static -engine cannot safely widen its sources to library APIs without reintroducing that false-alarm explosion** — -`banks` path: sink `path.open` found, `source_local/near=false`; `graphiti`/`bedrock` injection: no source. +false SQL-injections on WordPress/PMPro (documented in the `_parameters_are_input` docstring). The static +engine cannot safely widen its sources to library APIs without reintroducing those false alarms. +For `banks` path: sink `path.open` found, `source_local/near=false`; `graphiti`/`bedrock` injection: no source. -**This is the precise seam for the maintainer's "#2 guided by #1" directive.** The static taint cannot *safely* +The maintainer's "#2 guided by #1" directive addresses this gap. Static taint cannot *safely* decide which library parameters are trust boundaries; the **LLM can** reason about which public-API surface is attacker-reachable (intent, not pattern). So: -- **#1 (LLM) guides:** on a changed library region the engine ranks but cannot source, the LLM adjudicates whether a +- #1 (LLM) guides: on a changed library region the engine ranks but cannot source, the LLM adjudicates whether a parameter is an untrusted trust boundary and names the source→sink it believes exists (search-with-proof). -- **#2 (taint) traces + proves:** the engine then runs the flow from the LLM-named source to the known sink, - confirming (or refuting) with a traced path — a proof, not the LLM's say-so. Net: catch without the static FP - blowup, and every catch carries a dataflow witness. -- **Feedback:** where the LLM repeatedly names the same source shape (e.g. a registry-key → path family), promote it +- #2 (taint) traces + proves: the engine then runs the flow from the LLM-named source to the known sink, + confirming or refuting it with a traced path. The trace provides proof beyond the LLM's judgment. This would avoid the static FP + increase while giving every catch a dataflow witness. +- Where the LLM repeatedly names the same source shape (e.g. a registry-key → path family), promote it to a static fact in `python.toml` guarded by the entry/shape conditions that keep WordPress quiet. -**Proposed first increment (pending budget confirmation — LLM calls cost money):** wire the disabled LLM tier on the +Proposed first increment (pending budget confirmation, LLM calls cost money): wire the disabled LLM tier on the 5 pilot regions in *LLM-guided-source* mode and measure: (a) does it name the right trust-boundary source per vuln, (b) does the taint engine then trace source→sink and emit a vuln-family finding, (c) catch lift over regex (1/5) and false-alarm cost on the ordinary changes. Stage it (5–10 sizing → ~10–20 → full on lift). This makes "#2 guided by @@ -303,79 +303,78 @@ yields findings. Sized the LLM tier on banks (the clearest pilot miss) with the project's own DeepSeek client (`deepseek-flash`). -**Infra + cost (sizing answer):** DeepSeek reachable, model id `deepseek-flash` valid. **~$0.00016–0.00025 per -call, ~1.1 s latency.** Sub-cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback -(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param — a client fix (don't send +DeepSeek is reachable, and model id `deepseek-flash` is valid. **~$0.00016–0.00025 per +call, ~1.1 s latency.** This costs less than a cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback +(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param. A client fix (don't send `thinking` to that endpoint) is needed before it can be used; DeepSeek works today so it is not blocking. -**"2 guided by 1" validated as a mechanism — the LLM gives the correct verdict WHEN given the real sink + source -context:** -- Asked about `_load` (the pool's SZZ-labelled site, 135–141): `vulnerable=false` — *correct*, `_load` reconstructs +"2 guided by 1" works in this case: the LLM gives the correct verdict when given the real sink + source +context: +- For `_load` (the pool's SZZ-labelled site, 135–141), the model returned `vulnerable=false`. This is correct: `_load` reconstructs paths from its own index, not the sink. -- Asked about the REAL sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the - context that `prompt.name` can be request-derived: **`vulnerable=true`** — *"prompt.name is joined into a - filesystem path and written without sanitization… path traversal or arbitrary file write."* A correct catch. +- Asked about the real sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the + context that `prompt.name` can be request-derived: **`vulnerable=true`**, *"prompt.name is joined into a + filesystem path and written without sanitization… path traversal or arbitrary file write."* This is a correct catch. -**Two blockers to realising it in the live pipeline (both now concrete):** +Two blockers to realising it in the live pipeline (both now concrete): 1. **Context, not single-shot.** The model only catches with the real sink + caller/source context. A single-shot question on the ranker's region (which pointed at `_load`, and at the benign local-glob `_scan`) does not. This requires the **agentic search-with-proof hunter** (explores callers/entry points to establish the trust - boundary), not the single CANDIDATE_QUESTION path. That is more calls → the real cost driver to size next. -2. **Corpus-label soundness.** The pool's banks label is **mislocated**: `head_lines=[[30],[104],[138–141]]`, - `function=_load` — the real sink is line **44**, outside head_lines±10 and a different function. So (a) even a - correct finding at line 44 would NOT score as a catch under `location_match`, and (b) the sink pre-exists in base - (the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet** — the pool even + boundary), not the single CANDIDATE_QUESTION path. That needs more calls → their cost is the next measurement. +2. **Corpus-label soundness.** The pool's banks label is mislocated: `head_lines=[[30],[104],[138–141]]`, + `function=_load`. The real sink is line **44**, outside head_lines±10 and a different function. So (a) even a + correct finding at line 44 would not score as a catch under `location_match`, and (b) the sink pre-exists in base + (the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet**. The pool even annotates this bias ("delta hook can correctly stay quiet"). **Part of the pilot's engine 0/5 is this SZZ mislocation, not a detection failure.** -**Revised next steps (budget approved, stay staged):** +The budget is approved. Keep the revised next steps staged: - **Validate the 5 pilot labels first** (cheap, LLM-assisted corpus audit): for each, is the real sink inside the changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair targets. This separates true detection gaps from SZZ artifacts before spending on detection runs. -- **Then run the AGENTIC hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost; +- **Then run the agentic hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost; measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter is the real "#1". - Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2). ## Prompt evolution is a CORE lever for the LLM arm (maintainer directive 2026-10-12) -**"What is key is that we evolve the prompts as well"** — and **"GEPA or comparable; this is an evolving field."** -The bedrock hunter result makes this concrete: the agentic hunter ran (64 model calls, $0.0076) and reasoned -carefully, but with its **current prompts** it flagged adjacent concerns (network-PUBLIC default, download_file) -and never named the actual `install_packages` command injection — a *prompting/targeting* miss, not a cost or -capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence -outputs, evolves the prompt, and retries — built on **DSPy** (its GEPA / MIPRO optimisers), kept **pluggable** -behind an interface because the field moves fast (GEPA, MIPRO, TextGrad, successors). This aligns with -[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to BOTH LLM -uses here — the search-with-proof detector and the corpus-label auditor. +**"What is key is that we evolve the prompts as well"**, and **"GEPA or comparable; this is an evolving field."** +The bedrock hunter ran (64 model calls, $0.0076), but its **current prompts** led it to flag adjacent concerns +(network-PUBLIC default, download_file). It never named the `install_packages` command injection. This was a +*prompting/targeting* miss, rather than a cost or capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence +outputs, updates the prompt, and retries. Build it on **DSPy** (its GEPA / MIPRO optimisers). Keep it **pluggable** +behind an interface because the field changes quickly (GEPA, MIPRO, TextGrad, successors). This aligns with +[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to both LLM +uses here: the search-with-proof detector and the corpus-label auditor. -**The optimisation objective is to learn WHAT HINTS/ANNOTATIONS the model needs** to reach the real sink — not -just reword the prompt. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package +The optimisation must learn which hints/annotations the model needs to reach the real sink. Rewording the +prompt alone is insufficient. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package names → command injection"; banks needed "`prompt.name` is request-derived and joined into a filesystem path → traversal". DSPy optimises which evidence/hints to feed the signature so the model targets the true sink instead of adjacent noise. -**Source those hints from the framework knowledge the project ALREADY ships** — the semantic facts taxonomy -(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework WITHOUT +Use the project's existing framework knowledge for those hints. The semantic facts taxonomy +(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework without user annotations ([[php-detection-is-engine-only]] zero-annotation value). Where framework knowledge covers the -API, feed it as a hint; where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary -and, on repeated confirmation, it graduates into the facts (the #2 feedback loop). No new annotation burden on the -user. This is now a first-class part of the "#2 guided by #1" workstream, not a deferred nicety. (DeepSeek +API, feed it as a hint. Where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary, +which becomes a fact after repeated confirmation (the #2 feedback loop). No new annotation burden on the +user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek `deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.) ## POOL-P1 LABEL AUDIT (LLM, 2026-10-12) — 47% of labels are sound -Widened the banks label-mislocation check across all **49 public** pool vulns via DeepSeek (advisory + head code + +The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code + pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`. - **label_sound = 23/49 (47%)**; 53% unsound (25 pre-exist in base, 16 sink outside changed lines). Cost $0.038. - By family (sound/total): injection 10/13, access_control 5/8, untrusted_destination 4/7, deserialization 1/2, config_secrets 1/4, **path 1/7, output_encoding 1/8**. Injection labels mostly sound; path/output_encoding mostly mislocate or pre-exist. -- **record 07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook** — on the ~53% +- Record 07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook. On the ~53% pre-existing/mislocated vulns a change-scoped hook correctly stays quiet. The honest catchable denominator is the sound subset; record 07 needs this caveat + a sound-subset recomputation. -- **Fair pilot re-score:** only bedrock + graphiti (injection) of the 5 are sound; the engine missed both — the - real detection gaps. The 23 sound targets are the fair local eval set for "#2 guided by #1". +- Only bedrock + graphiti (injection) of the 5 pilot cases have sound labels. The engine missed both, so these + are detection gaps. The 23 sound targets are the fair local eval set for "#2 guided by #1". - Next: run the agentic search-with-proof hunter on the sound targets (start bedrock + graphiti), size its per-vuln call count/cost, measure catch + false-alarm. Then recompute record 07 on the sound subset. @@ -384,13 +383,13 @@ pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-1 Ran `ousast scan --mode deep` (DeepSeek `deepseek-flash`, model layer + tool_hunter, 60-call/8-hotspot bound) on bedrock + graphiti, the two sound pilot targets the static engine missed. Artifact: `benchmarks/measurements/2026-10-12-hunter-sound-targets/`. -- **hunter 1/2 vs static engine 0/2.** graphiti: **HIT** — `falkordb_driver.py:207`, tag `family:cypher-injection`, +- **hunter 1/2 vs static engine 0/2.** graphiti: **HIT**, `falkordb_driver.py:207`, tag `family:cypher-injection`, rationale traces `build_fulltext_query` concatenating user input into a Cypher string at line 278 (inside the - labelled function, location-matches). bedrock: **miss** — 12 suspicions on adjacent concerns, never named the + labelled function, location-matches). bedrock: **miss**, 12 suspicions on adjacent concerns, never named the `install_packages`→`pip` command injection. -- **Cost is a non-issue:** ~$0.007/repo, 64 model calls for a whole-repo deep scan. -- **Everything stopped at `suspicion`** — the proof arm did not elevate to corroborated/entailed/execution_confirmed. - Release-grade BLOCK needs *proven* findings; measuring the proof arm's lift is the next depth. +- Cost is low: ~$0.007/repo, 64 model calls for a whole-repo deep scan. +- Everything stopped at `suspicion`. The proof arm did not elevate to corroborated/entailed/execution_confirmed. + Release-grade BLOCK needs *proven* findings; the next step is to measure the proof arm's lift. - bedrock's miss is the concrete case for the DSPy prompt/hints loop above. Next: (a) measure the proof arm (can it elevate graphiti's suspicion to proven?); (b) stand up the DSPy hint loop @@ -399,37 +398,37 @@ run for the release-relevant shape. ## STRATEGIC PIVOT: core problem = unprovable reachability; FIX THE EVALUATION TARGET FIRST (2026-10-12) -**Core problem (named with the maintainer):** the tool can SPOT dangerous ops but cannot prove they are -ATTACKER-REACHABLE — the trust boundary lives outside the changed library code, in the caller/application a -change-scoped scan does not see; and the project rightly won't emit unproven alarms. Recall → ~0 from unprovable -reachability, not blindness. Cost/latency/engine-wall/corpus-labels were all downstream noise. +The maintainer and team identified the problem: the tool can find dangerous operations but cannot prove +an attacker can reach them. The trust boundary is outside the changed library code, in a caller/application +that a change-scoped scan does not see. The project will not emit unproven alarms. Recall → ~0 from unprovable +reachability, not blindness. Cost/latency/engine-wall/corpus-labels did not explain this problem. -**Chosen bet (of three — fix-target / advisory-with-conditions / build-prover): FIX THE EVALUATION TARGET FIRST.** -A pre-push hook runs inside a developer's APPLICATION (entry points present, request→sink in-repo), but the pool is +The chosen bet is to fix the evaluation target first. The three options were fix-target / advisory-with-conditions / build-prover. +A pre-push hook runs inside a developer's application (entry points present, request→sink in-repo), but the pool is library-heavy, so we measured the hardest/wrong case. -**Shape classification (LLM, 49 public vulns, $0.033 — `benchmarks/measurements/2026-10-12-pool-p1-shape-classification/`):** -reachability_in_repo True only 12/49 (24%); 37/49 are pure libraries. A legitimate catch needs sound label AND -reachability in-repo → **5/49 (~10%)**: graphiti, desktopcommandermcp, malla, document-merge-service, mcp-remote. -**Shape predicted the hunter result** (graphiti app-shaped→caught; bedrock library→missed). So record-07's 3/92 is -scored against a ~90%-illegitimate denominator — the near-zero catch is largely a TARGET MISMATCH. +Shape classification (LLM, 49 public vulns, $0.033, `benchmarks/measurements/2026-10-12-pool-p1-shape-classification/`): +reachability_in_repo True only 12/49 (24%); 37/49 are pure libraries. A valid catch needs a sound label and +in-repo reachability → 5/49 (~10%): graphiti, desktopcommandermcp, malla, document-merge-service, mcp-remote. +Repository shape predicted the hunter result (graphiti app-shaped→caught; bedrock library→missed). So record-07's 3/92 is +scored against a ~90%-illegitimate denominator. The near-zero catch largely comes from a mismatch between the targets and the tool. -**Next:** deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch rate on the corrected set; then +Next: deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch rate on the corrected set; then stand up an app-shaped eval set from known-vulnerable apps (VAmPI/NodeGoat/DjanGoat/vibe-py) and recompute the -floor. Deferred bets (advisory-with-conditions, reachability prover) revisited once the fair-target numbers land. +floor. Revisit the deferred bets (advisory-with-conditions, reachability prover) once the fair-target results are available. ## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy) — target-fix validated (2026-10-12) Deep scan on the 5 app-shaped+sound targets (`benchmarks/measurements/2026-10-12-hunter-fair-targets/`): -- **3/5 detected**: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE - overlay — "outbound.request reached from parameter" on readFileFromUrl; the param-as-source mode fired because an - entry point is present). **2/5 missed**: malla (DOM/template XSS) and dms (Jinja SSTI) — both in families - families.toml DOCUMENTS as limited (template-mediated escaping). -- vs the library-heavy pilot's ~0 → **the target mismatch was real and material.** On an app-shaped repo the engine +- 3/5 detected: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE + overlay, "outbound.request reached from parameter" on readFileFromUrl; the param-as-source mode fired because an + entry point is present). 2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families + documented as limited in families.toml (template-mediated escaping). +- The library-heavy pilot found ~0 → this confirms that the target mismatch affected detection. On an app-shaped repo the engine itself detects (parameter is a recognized source once the entry point is in-repo). -- **But everything stops at `suspicion`/`static_corroboration`** — none proven. Cost ~$0.007/repo. +- Everything stops at `suspicion`/`static_corroboration`. None are proven. Cost ~$0.007/repo. -**Strategy verdict:** bet #1 (fix the eval target) is validated as NECESSARY and MATERIAL but NOT SUFFICIENT. +Bet #1 (fix the eval target) is necessary and improves detection, but is insufficient on its own. Detection appears on fair targets; release-grade BLOCK still needs (a) proof-arm elevation (suspicion→proven) and (b) family coverage (template XSS/SSTI) + the DSPy hint loop for misses. Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure the proof arm's suspicion→proven lift. @@ -438,53 +437,53 @@ sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure Measured whether the proof arm can elevate the 3 fair-target suspicions (graphiti, mcp-remote, dcmcp) to `exploit_demonstrated`. -- **Correction:** the search arm's `FAMILY_ORACLES` (search/demo.py) differs from families.toml's `verifier`. - All 3 hits HAVE a runtime oracle: injection→(sql,command), untrusted_destination→(ssrf). (deserialization/ - access_control/config_secrets have NONE — empty tuple → `no_oracle`.) So dcmcp IS provable in principle +- The search arm's `FAMILY_ORACLES` (search/demo.py) differs from families.toml's `verifier`. + All 3 hits have a runtime oracle: injection→(sql,command), untrusted_destination→(ssrf). (deserialization/ + access_control/config_secrets have none, empty tuple → `no_oracle`.) So dcmcp is provable in principle (earlier "unprovable" was a families.toml misread). -- **But proving is a runtime differential:** `search/verify.verify()` requires (1) an AUTHORED declarative demo +- Runtime differential proof through `search/verify.verify()` requires (1) an authored declarative demo (start the app on a port, run exploit steps, fill oracle placeholders: sql→database_path, command→marker_dir, - ssrf→callback_url), (2) BOTH head AND base runnable with OFFLINE deps (`pip --no-index`, `npm ci --offline`), + ssrf→callback_url), (2) both head and base runnable with offline deps (`pip --no-index`, `npm ci --offline`), (3) a sandbox differential "owned effect in 3/3 affected runs and 0/3 safe runs". Local host has a sandbox (`isolation_mode: userns`); the plane supplies a task-boundary dispatcher. -- **The scan/push path reaches only `suspicion`/`static_corroboration`** (verification.py levels) — there is NO code +- The scan/push path reaches only `suspicion`/`static_corroboration` (verification.py levels). There is no code path from a scan suspicion to `exploit_demonstrated`, and no caller wires the search Coordinator from the push - flow. So **proof-arm lift at push time = 0/3**: not because the vulns are unreal, but because proving is a - per-target, offline-build, runtime, differential operation that cannot run inside a pre-push hook. (Prior - per-pair demo attempts were mostly inconclusive on this exact wiring — app startup, offline deps, oracle plumbing.) + flow. So proof-arm lift at push time = 0/3: not because the vulns are unreal, but because proving is a + runtime differential operation with an offline build for each target that cannot run inside a pre-push hook. (Prior + per-pair demo attempts were mostly inconclusive on this exact wiring, app startup, offline deps, oracle plumbing.) -**Strategic consequence (resolves the three-bet fork):** proof/BLOCK is an ASYNC capability (CI / plane / triage -with authored demos + a runnable sandbox), NOT a pre-push one. At push time the honest ceiling is corroborated -suspicion. So: -- **Bet #2 (advisory-with-conditions) is the honest PRE-PUSH product**: surface corroborated suspicions ("change +This resolves the three-bet choice. Proof/BLOCK runs asynchronously (CI / plane / triage +with authored demos + a runnable sandbox). At push time, the strongest available evidence is corroborated +suspicion: +- Bet #2 (advisory-with-conditions) is the honest PRE-PUSH product: surface corroborated suspicions ("change introduces a flow that looks like X; reachable if untrusted Y reaches it"), let the dev / coding-agent (who has - the app context) judge — matching the "oversee coding agents" framing and the suspicion-rung reality. -- **Bet #3 (the proof arm) is the ASYNC BLOCK lane**: run `search/verify` on the plane/CI for the suspicions worth - proving, with authored (eventually DSPy-authored) demos. Worth an empirical run on one tractable target - (mcp-remote) to confirm the arm demonstrates on a real unseen vuln — a bounded, separate effort. + the app context) judge. This matches the "oversee coding agents" purpose and the evidence available at the suspicion rung. +- Bet #3 (the proof arm) is the ASYNC BLOCK lane: run `search/verify` on the plane/CI for the suspicions worth + proving, with authored (eventually DSPy-authored) demos. Run a bounded, separate test on one tractable target + (mcp-remote) to confirm that the arm demonstrates a real unseen vuln. ## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave -the engine ZERO path findings in the pilot). Same target, same CPG — only the search changes. +no engine path findings in the pilot). The target and CPG are unchanged; only the search changes. - Injected the two annotations a per-target LLM steer would supply: **parameter-as-source** (`parameter_sources=True`, so `prompt.name` counts as untrusted) + the **missing file-write sink** (`write_text`/`read_text` added to path sinks). -- **Baseline (shipped search): 0 resolved `prompt → write_text` flows** — its 5 flow rows are all census/ +- The shipped search baseline resolved 0 `prompt → write_text` flows. Its 5 flow rows are all census/ evidence (`sink=None`), matching the pilot's zero path findings. -- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in - `from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear ONLY with +- Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in + `from_prompt_path` (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear only with the annotations. -- **Thesis demonstrated on a concrete case:** banks was not a wrong target; the search was unannotated. The - fix needed exactly **two** inputs — the param-source and the file-write sink — which are precisely what the +- The banks result supports the hypothesis: the target was valid, but the search lacked annotations. The + fix needed exactly **two** inputs: the param-source and the file-write sink. These are what the per-target LLM steer (DSPy-optimised, framework-seeded) has to learn to supply. This is the first measured - instance of the "#2 guided by #1" loop closing on an unseen target. -- Caveats: (1) one row shows `source=path` not `prompt` — the param-source fires broadly, a precision knob for - the steering loop; (2) this is a TRACE, not the runtime differential proof (proof stays the async BLOCK lane). + instance of a complete "#2 guided by #1" loop on an unseen target. +- Caveats: (1) one row shows `source=path` not `prompt`, the param-source fires broadly, the steering loop needs to control that breadth to improve + precision; (2) this is a trace, not the runtime differential proof (proof stays the async BLOCK lane). -**Instrument note / filed bug:** the first MVP used the single-query `cpg.run("taint", …)` path and returned -0/0 — an INSTRUMENT FAILURE, not a result (`cpg query taint failed` on every query). Root cause filed in +The first MVP used the single-query `cpg.run("taint", …)` path and returned +0/0 because of an instrument failure, rather than producing a valid result (`cpg query taint failed` on every query). Root cause filed in `bugs/taint-sc-single-query-signature-skew.md`: `request_params()` emits 5 keys (`guards`, `fixedOrigin`, `originAnchors`, `quotedSanitizers`, `fieldParameterSources`) that `taint.sc`'s `exec` signature does not declare, so both single-query sub-routes (CLI `--param` and session named-arg) reject them. The batch path diff --git a/.kiro/specs/valid-detection-benchmark/brief.md b/.kiro/specs/valid-detection-benchmark/brief.md new file mode 100644 index 0000000..9524546 --- /dev/null +++ b/.kiro/specs/valid-detection-benchmark/brief.md @@ -0,0 +1,85 @@ +# Valid detection benchmark + +Created 2026-10-12. This brief **supersedes the frozen-SZZ-pool evaluation** (`unseen-repo-evaluation`), which is retired. +SZZ guessed its labels and placed the vulnerability in the wrong location about half the time (audit: path 1/7, +output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Catch / +false-alarm numbers from that pool are not valid benchmark results. This brief defines a benchmark with +sound ground truth and metrics that engineers and managers can understand. + +## The two questions the benchmark answers + +A pre-push security check must answer two questions: + +1. **If I'm about to push a real vulnerability, does it stop me?** +2. **If my change is fine, does it leave me alone?** + +The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on, +as is the LLM-guided arm. The repositories are ones the tool has **never seen**. + +## What makes the data valid + +### Positive cases — "should catch" (real, verified, reachable vulnerabilities) +- Take the label from the security fix, never from SZZ. Use the code *just before* the fix, at the file/function + the fix changed. The fix establishes the vulnerability location without a blame walk-back or mislocation. +- Keep only app-shaped cases where the vulnerability is reachable from an entry point *inside the repo*. This + can be a web route or a request handler; a CLI argument also qualifies. Drop library-internal vulnerabilities + that depend on a caller outside the repo. A pre-push tool cannot see that trust boundary, so scoring it is unfair. +- Verify each case through LLM adjudication and a human spot-check. Confirm (a) that the pre-fix code is + vulnerable and (b) that the fix addresses it. Also confirm (c) that it is reachable in-repo. + +### Negative cases — "should not alarm" (ordinary safe changes) +- Use everyday commits from the **same** application repositories that are not security fixes. They are plentiful + and cheap to collect. Almost every real push is ordinary, so false alarms determine whether developers keep + using the tool. The negative set must be much larger than the positive set. + +### Held out +- None of these repositories were used to build, train, tune, mine or harvest the tool. + +A few dozen vulnerabilities with sound labels and in-repo reachability, each verified, are better than hundreds +of SZZ guesses. SZZ is wrong about half the time. Curating this set takes manual work, supported by the LLM-audit +tooling that found the SZZ pool's defects. Dozens of good cases are enough. + +## Metrics someone understands + +| Metric | Plain-English question | Example | +|---|---|---| +| **Catch rate** (recall) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" | +| **False-alarm rate** | On safe changes, how often does it cry wolf? | "Flagged 3 of 100 ordinary pushes" | +| **Precision** | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" | +| **Proof rate** | Of what it flags, how much can it *demonstrate*, not just suspect? | "12 of 18 catches came with a working demonstration" | + +Report false alarms as **"per 100 pushes"**, which reflects a developer's experience. Vulnerabilities are rare +among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it. + +Report the first two metrics together: + +> "On unseen application repos: catches 90% of real vulnerabilities while false-alarming on 3% of ordinary +> pushes." + +This wording is easier for a manager to understand than "3/92, Wilson95 1.1–9.2%". + +## The gold-standard metric — and the project's edge + +**Proved catches** do not depend on labels. The search-with-proof / differential-exploit arm demonstrates the +bad effect on the vulnerable code and its absence on the fixed code. The demonstration *is* the ground truth. +The headline to aim for is: + +> "On unseen app repos: caught and *proved* N of M real vulnerabilities, with K false alarms per 100 ordinary +> pushes." + +The architecture is built to produce this kind of benchmark. + +## Why the old approach was not feasible (recorded so it is not re-tried) + +- SZZ ground truth is machine-guessed and ~half wrong → every catch/FP number off it is noise. +- Library-heavy corpus → most cases are unreachable in-repo, so a pre-push tool is scored on things it + structurally cannot see. +- Scoring regex-only (engine and LLM off) measured a tier nobody would ship alone. +- Automated harvesting prioritized pool size over sound ground truth and in-repo reachability. + +## Open questions for the spec phase + +- How many verified positives are enough for a credible catch-rate interval (target set size)? +- The negative:positive ratio that reflects real push traffic (for an honest false-alarms-per-100-pushes). +- Curation workflow: how LLM adjudication + human spot-check are recorded and frozen per case. +- Language/framework coverage of the positive set (be explicit about what is and isn't represented). diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 18d48c0..9e16605 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -7,23 +7,23 @@ This section supersedes the ordering below it. The older sections stay as histor ### Destination On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every -family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work -is split by what each place is good at: +family, and BLOCK only for families whose precision has been proven on repositories the system never saw. +Each environment handles different parts of the work: -- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks. +- The kube-ax cluster (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes) runs the Joern engine as AX Tasks. No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links. -- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store, - and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap. +- The VM or CI runner handles candidate selection and the decision engine's model calls (DeepSeek). It also hosts + the memory store and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap. - **kind** stays the local development profile, with the same code and manifests. ### The question everything answers -Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every -phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide -whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it: +The A/B experiments test whether the safety net works on repositories it has never seen. Every phase below must +answer that question. Almost every push on an unseen repository is an ordinary change. Two numbers measure whether +the safety net is useful, based on the hook's advisory output as a user sees it: -1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change. -2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything. +1. How often does it flag real vulnerability-introducing changes in unseen repositories (catch rate)? +2. How often does it flag anything on ordinary changes in the same unseen repositories (false-alarm rate)? Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal. @@ -37,17 +37,19 @@ Within-pair AUC on known fix pairs answers neither. It stays a development signa | Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` | | Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` | -Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did -not move a generalisation number. From here, infrastructure and engine work is done only when it removes a -bottleneck for the measurements below. +Correction recorded 2026-10-04: most work in the previous two days went into infrastructure and engine internals +without improving a generalisation number. Further work on those areas must remove a bottleneck for the +measurements below. ### Where we are (2026-10-11) -The G0 measurement is **done**: the unseen pool is **frozen**, the pipeline is hardened and **bucket-first**, -and the slice-1 baseline (record 07) ran on the **docker lane** — the sanctioned interim while the kube `ax` -lane is restored. G0 is exited; the kube `ax` lane is now a **G1** concern (the engine arm at scale), not a -baseline blocker. (Correction 2026-10-12: an earlier revision of this section said the baseline was "the one -remaining step, to run on kube" — that was stale; record 07 was completed on docker right after, see below.) +**The frozen-SZZ-pool scoring approach is not feasible. Its catch/false-alarm numbers are retired as a +metric (2026-10-12 maintainer call).** The SZZ-derived labels place the vulnerability in the wrong location +~half the time (audit: path 1/7, output_encoding 1/8 sound). Only ~24% of entries are app-shaped +(reachability in-repo). Catch/false-alarm figures from this pool are invalid as a baseline. +Record 07's "3/92" and every pool catch/FP number are **not quoted and not a release gate**. Before any +detection number is valid, the pool must have **sound labels** and entries classified as app-shaped or library. +Until then, no pool-scored metric is cited anywhere. | Area | State | Evidence | | --- | --- | --- | @@ -56,25 +58,21 @@ remaining step, to run on kube" — that was stale; record 07 was completed on d | Harvest | Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop `--filter=blob:none`, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; **GitHub-rate-limit bound, not CPU**. | 95bed70, 857263d, 21aebe8, 7b711a6 | | Persistence | **Bucket-first, not local disk.** `ousast-corpus` is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and **replay outputs go to RustFS too**, not `~/ousast-results` on local disk. | `storage-streaming.md`, `bank.py` | | Replay image | Provenance pinned. CI `engine-image.yml` builds `ousast-replay-task` (harness with `/app/benchmarks/unseen/replay_entry.py`) + `ousast-engine-task` (Joern hook), publicly pullable by digest. The old `ousast-runner:dev` (248 MB ax-runner, no `/app/benchmarks`, no Joern) is the **wrong** image for replay. | CI run 38116281676 | -| Baseline (G0.2) | **DONE — record 07 (`e901640`), docker lane.** 1300/1300 changes; catch 3/92 = **3.3%** (Wilson95 1.1–9.2%), false-alarm **3.1%** (cluster95 1.6–5.4%), instrument_failure 54 (4.2% < 5% guard). `engine_covered=0` and no model calls → this is a **regex/quick-rule floor** (the engine never ran under the ~200s pre-push wall; the LLM tier was disabled). Baseline row in `usage-p1.jsonl` (freeze_digest `3371c67d`) = **G0 exit**. Ran on docker (not kube `ax`) per the sanctioned interim. | `benchmarks/measurements/2026-10-11-unseen-07-slice1-baseline/` | +| Baseline (G0.2) | **RETIRED as a metric.** A replay (record 07) ran end-to-end on the docker lane, which proves the pipeline works — but its catch/false-alarm figures are **not a usable baseline** (scored against ~half-mislocated SZZ labels on a mostly non-app-shaped corpus) and are **not quoted or used as a gate**. The pool must be cleaned to sound labels first. | pipeline proven; metric not used | -Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives -in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next -pool or arm is cheap, not a rediscovery. +Sourcing and engine-image work took time. The pipeline and these docs now record bucket-first persistence and +pinned image provenance, along with the measured yield. The next pool or arm can reuse that work. -### Readiness plan (G0 exited; next is G1) +### Readiness plan — build a VALID benchmark first -**G0 exit = record 07 is DONE** (docker lane, regex/quick-rule floor; see the Baseline row above). Steps 1–4 -below are complete. The kube `ax` lane was NOT needed for the baseline and is now a **G1** target. +The frozen-SZZ-pool approach is retired (above). The replay pipeline works end-to-end, and the harvest/freeze/bucket +machinery remains available. The engine/replay images are pinned. No catch/false-alarm figure from that pool is used. -1. ~~Kube replay lane~~ — **not needed for G0**; the baseline ran on the docker lane (the sanctioned interim). - The `ax` lane moves to G1 (see below). -2. ~~Unblock the replay~~ — done; replay ran over the frozen slice. -3. ~~Run slice-1 baseline~~ — done: `--lane docker --arm default`, 1,300 changes, no model cost, outputs to RustFS. -4. ~~Adjudicate (7.3) → record 07 (7.4)~~ — done = **G0 exit** (`usage-p1.jsonl` freeze_digest `3371c67d`). -5. **Next — G1 arms on the frozen pool.** Re-score the arms that the regex floor cannot be (engine evidence, - DeepSeek decision engine, combiner) to see what beats catch 3.3% at a fixed false-alarm budget. This is where - the kube `ax` lane is needed, because the engine arm cannot run under the ~200s pre-push wall on a single host. +Next, define and build the valid benchmark. Use sound, verified positives with in-repo reachability and ordinary +negatives on unseen repos. Measure the whole tool with metrics an engineer understands. The design is in +`.kiro/specs/valid-detection-benchmark/brief.md`. Only once that benchmark exists does any detection number +(catch rate, false-alarms-per-100-pushes, proved-catch) mean anything. The kube `ax` lane is still the place the +engine tier runs at scale, but it is infrastructure, not the gate. ### Restoring the kube `ax` lane (for G1, operator-owned) @@ -82,10 +80,10 @@ The AX executor is **built and smoke-tested** (`benchmarks/ax/batch.py`, `engine lanes, OOM→docker fallback; an engine smoke test dispatched to the 2-worker cluster end-to-end on 2026-10-04). What remains before the G1 engine-arm replay: -- **Use the working `ax` binary: `~/go/bin/ax`** — NOT `~/.local/ax-v0.3.1/ax`, which hits the AX-lane blocker +- Use the working `ax` binary, `~/go/bin/ax`. `~/.local/ax-v0.3.1/ax` hits the AX-lane blocker (recorded in the unseen spec `tasks.md`, 2026-10-11). The coordinator commands in the READMEs are being corrected to this binary. -- **Store endpoint:** `benchmarks/ax/batch.py` expects a public-HTTPS DNS, presign-capable S3 endpoint reachable +- `benchmarks/ax/batch.py` expects a public-HTTPS DNS, presign-capable S3 store endpoint reachable from the cluster; the cluster carries no model/store credentials (it reads/writes via short-lived presigned links). This and the cluster's store access are operator/AX-owned. - **Run the open spec tasks** that gate G1: **4.4** (engine smoke on kube-ax → record 04) and **5.4** (replay @@ -94,23 +92,27 @@ What remains before the G1 engine-arm replay: ### Phases -**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`) +**G0. A VALID detection benchmark** (spec `valid-detection-benchmark`; supersedes the retired +`unseen-repo-evaluation` frozen-SZZ-pool approach) -1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population. - Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where - possible) and ordinary commits from the same history. -2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and - record catch rate and false-alarm rate, with confidence intervals, per family and overall. -3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one. +1. Use real vulnerabilities labelled *from the fix*: the pre-fix code at the file/function the fix changed, + without an SZZ blame-walk. Keep only those reachable from an in-repo entry point. Confirm each case through + LLM adjudication and a human spot-check so the positives are sound, app-shaped and verified. +2. Ordinary negatives: everyday non-fix commits from the same application repos, a set much larger than the + positives (false alarms discourage adoption). +3. Measure the **whole tool** (all tiers on) on **unseen** repos, and report metrics an engineer reads: catch + rate, **false-alarms per 100 pushes**, precision, and **proved-catch** (demonstrated, label-free). See the + brief for the full definition and why the SZZ pool was not feasible. -Exit: the pool is frozen and the current production hook has a measured baseline on it. +Exit: the benchmark exists (frozen, verified) and the whole tool has a first measured result on it. -**G1. Re-score the arms we have on the pool** +**G1. Score the arms on the valid benchmark** -Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is -an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget. +Quick rules, engine evidence, the LLM-guided arm, and their combination, each a registered decision rule, +scored by catch rate at a fixed false-alarms-per-100-pushes budget, with proved-catch where available. -Exit: one measured table of arms on unseen code; keep only what generalises. +Exit: one measured table of arms on unseen app-shaped code; keep only what generalises. (No number is quoted +until it comes from this benchmark, never the SZZ pool.) **G2. Improve what moves the unseen-repo numbers** @@ -120,63 +122,63 @@ changes, more labelled pairs from more distinct repositories. Active G2 workstreams (2026-10-11; progress tracked in `.kiro/specs/engine-change-scoped-analysis/brief.md`): -**#1 priority after record 07: get real (semantic) detection under the pre-push budget.** The record-07 baseline -is **regex only** — verified: the push arms hardcode `model="disabled"`, `max_model_calls=0`, and the engine never -ran (`engine_covered=0`); measured `catch≈2.5%`. Pattern rules alone will never be enough (else `grep` would do). -The two tiers that carry detection both sat out: the **engine** (Joern dataflow) and the **LLM adjudication** -(disabled in the push path). The workstream is an **efficiency barrier analysis** — a per-tier budget ledger +Priority #1 after record 07: run semantic detection within the pre-push budget. The record-07 baseline +is **regex only**. The push arms hardcode `model="disabled"`, `max_model_calls=0`, and the engine never +ran (`engine_covered=0`); measured `catch≈2.5%`. Pattern rules alone are insufficient; otherwise `grep` would do. +The two detection tiers did not run: the **engine** (Joern dataflow) and the **LLM adjudication** +(disabled in the push path). The workstream measures **efficiency barriers** with a per-tier budget ledger against a realistic pre-push window (≤120 s target, 300 s hard): - - **Engine: E1/E2 DONE (2026-10-12, clean host) — see brief "E1/E2 RESULTS".** CORRECTION: the engine's + - Engine: E1/E2 DONE (2026-10-12, clean host), see brief "E1/E2 RESULTS". CORRECTION: the engine's `query=0` wall was **host contention** (the baseline replay), not an intrinsic floor. On a clean host the shipped engine **runs dataflow** (query 0→35 s, selected 0→**356 regions**). The evidence/taint batch (~225 s) is **invariant to session and preload**; graph-load is cheap (~35 s incl `importCpg`). The real barrier is `taint.sc`'s ~160 s per-batch flow work, so `build(~30)+evidence(~225)≈255 s` of 300 s defers 1009/1365 - regions. Lever = **change-scope the family/file set** the taint batch walks (NOT session/preload). E3 analysis + regions. Reduce the work by **scoping the family/file set to the change** (not session/preload). E3 analysis per-pair cost; E4 build vs repo size remain. - - **ENGINE-ON PILOT DONE (2026-10-12, clean host, 5 public repos) — see brief "ENGINE-ON PILOT RESULTS".** - **regex 1/5, engine 0/5.** The engine added NO catches over the regex floor. The **budget hypothesis is - REFUTED**: 3/5 repos finished their queue (deferred=0) and still emitted no finding — so change-scoping is NOT - the path to catch. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser, - launch-editor, score 2.0), but **withholds any finding it cannot anchor 1:1 to changed lines**: all 5 hit + - **ENGINE-ON PILOT DONE (2026-10-12, clean host, 5 public repos), see brief "ENGINE-ON PILOT RESULTS".** + **regex 1/5, engine 0/5.** The engine added no catches over the regex floor. The budget hypothesis is + refuted: 3/5 repos finished their queue (deferred=0) and still emitted no finding, so change-scoping will not + improve catch rate. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser, + launch-editor, score 2.0), but withholds any finding it cannot anchor 1:1 to changed lines: all 5 hit `ambiguous_line_correspondence` → `witness_unresolved` → `coverage_status=incomplete`, suppressing even the - scored sinks. Confirmed NOT a shallow-clone artifact. **Release read: the net is NOT releasable on detection.** - - **CORRECTED after inspecting `head.findings`:** the engine produced **ZERO vuln-family findings** (path/ - injection/deser) on all 5 — only config_secrets. The tier1/score2.0 "sink" was the RANKER's static candidate, - not a taint finding. `ambiguous_line_correspondence` is a downstream red herring (nothing to anchor), so the + scored sinks. This is not a shallow-clone artifact. Detection does not meet release requirements. + - Corrected after inspecting `head.findings`: the engine produced no vuln-family findings (path/ + injection/deser) on all 5, only config_secrets. The tier1/score2.0 "sink" was the ranker's static candidate, + not a taint finding. `ambiguous_line_correspondence` does not explain the missing findings (nothing to anchor), so the region-level witness fix (old "Option A") was **refuted before implementation**. The **LLM tier was disabled** - (model_calls=0). **ROOT CAUSE:** pilot repos are LIBRARIES whose untrusted input is public-API parameters; + (model_calls=0). The cause is that pilot repos are libraries whose untrusted input is public-API parameters; Python taint sources are web/CLI only, and the param-as-source mode fires only for entry points (vuln regions - are `entry:false`) — widening it statically would reintroduce the WordPress/PMPro false-SQL-injection flood. - **Workstream = maintainer's "#2 guided by #1":** the LLM adjudicates which library parameter is a trust boundary + are `entry:false`). Widening it statically would reintroduce the WordPress/PMPro false-SQL-injection flood. + The workstream follows the maintainer's "#2 guided by #1" directive: the LLM adjudicates which library parameter is a trust boundary (static rules can't without FPs), the taint engine then traces that source→sink as proof, and repeated source - shapes get promoted to guarded static facts. First increment (PENDING BUDGET): wire the disabled LLM tier on the + shapes get promoted to guarded static facts. First increment (pending budget): wire the disabled LLM tier on the 5 pilot regions in LLM-guided-source mode; measure source naming, traced finding, catch lift vs regex 1/5, and false-alarm on ordinary changes; stage 5–10 → ~10–20 → full on lift. #3 change-anchoring deprioritised until tiers yield findings. - - **BANKS FIXED-SEARCH MVP DONE (2026-10-12) — "#2 guided by #1" closes on an unseen target.** See brief + - **BANKS FIXED-SEARCH MVP DONE (2026-10-12), "#2 guided by #1" closes on an unseen target.** See brief "BANKS FIXED-SEARCH MVP" + `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. On banks (the library that gave the engine zero), injecting the two annotations an LLM steer would supply (param-as-source + `write_text`/`read_text` sink) makes the engine resolve `prompt → path-join → write_text` where the - shipped search resolved nothing. Proves the governing principle "no wrong targets, only wrong searching" and - names the exact inputs the DSPy-optimised steer must learn. Filed instrument bug: + shipped search resolved nothing. This supports the principle "no wrong targets, only wrong searching" and + identifies the inputs the DSPy-optimised steer must learn. Filed instrument bug: `.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md` (single-query taint path dead on an exec-signature skew; batch path unaffected). - - LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool. + - LLM: L1 model latency and calls per push; L2 the recall lift of the LLM tier over regex+dataflow on the pool. - Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config. -- **Change-scoped engine analysis — DEMOTED again by the pilot.** Branch `engine-change-scoped-evidence-pass` +- The pilot lowers the priority of change-scoped engine analysis again. Branch `engine-change-scoped-evidence-pass` (unmerged). Fitting the budget does not produce catches (3/5 pilot repos finished the queue with zero findings), - so this is not on the catch critical path. Keep only as a latency optimisation once detection works. + so this is not required to improve catch rate. Keep only as a latency optimisation once detection works. - **Line-correspondence (merged c39fbb8).** `ambiguous_line_correspondence` fires on 5/5 realistic diffs, but the - pilot's corrected diagnosis shows it is NOT the detection blocker here — the static tiers produce no vuln-family - findings to anchor in the first place. A region-level witness fallback only matters once #1/#2 yield findings. + pilot's corrected diagnosis shows it does not block detection here. The static tiers produce no vuln-family + findings to anchor. A region-level witness fallback only matters once #1/#2 yield findings. **G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4). **G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`, engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept. -**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout, +G5. Qualification and rollout: adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout, BLOCK only for families that passed. ### Cross-cutting rules @@ -214,13 +216,13 @@ progress, not adopted). See `overview.md` for the current-state summary. Build an actionable pre-push security safety net for AI-accelerated development. The maintainer accepted the following order: **prove one useful detection, make it -fast, prove cross-language transfer, qualify independently, then enable narrow advisory +fast, prove cross-language transfer, qualify independently, then enable a narrow advisory rollout**. The executable outcome contract lives in the existing [pre-push-safety-net release milestone spec](../specs/pre-push-safety-net/release-milestones.md). **We are at M1b. M1a is met; no other release milestone is verified yet.** The published -`v1.2.0-alpha.1` and 28/29 completed implementation tasks describe delivered machinery, -not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO. +`v1.2.0-alpha.1` and 28/29 completed implementation tasks show that the machinery is implemented. +They do not demonstrate a useful safety net. Task 8.3 and rollout remain NO-GO. | Order | Milestone | Observable exit | Current state | |---|---|---|---| @@ -252,26 +254,26 @@ not dropped safety checks. Advisory and blocking share the same actionability ba Graph persistence now works: identical-tip replays take 18.5–18.9s. All 18 other measured transactions still time out and 0/24 target checks complete. A local Joern session census -probe is promising, but full-query equivalence and hook performance are unverified. +probe shows promise, but full-query equivalence and hook performance are unverified. These findings inform M2; they do not complete M1. No second IR/engine, new heuristic scope selector, differential graph mutation or unrelated backlog is part of this path. ## Existing Spec Updates -- [ ] pre-push-safety-net — Execute the accepted M1–M5 release milestone amendment. +- [ ] pre-push-safety-net, Execute the accepted M1–M5 release milestone amendment. Owns immutable replay, comparison, admission, evaluation and reporting. Preserve the existing completed tasks; task 8.3 remains blocked until qualification passes. -- [ ] contributor-scan — Supply the required query/operation/context fixes for M1, +- [ ] contributor-scan, Supply the required query/operation/context fixes for M1, followed by equivalent bounded session execution for M2 and shared transfer evidence. Frontend retention 2.18 and named census repair 2.19 are already verified. -- [ ] flow-aware-ranking — Preserve ranker-owned scope and normalized cross-language +- [ ] flow-aware-ranking, Preserve ranker-owned scope and normalized cross-language behavior. New ranking research is not a prerequisite for the first useful comparison. -- [ ] zero-with-a-reason — Adjacent diagnostics work; finishing its whole backlog is +- [ ] zero-with-a-reason, Adjacent diagnostics work; finishing its whole backlog is not required before M1. ## Specs (dependency order) -- [ ] pre-push-safety-net — Existing approved implementation, dependent on contributor-scan. +- [ ] pre-push-safety-net, Existing approved implementation, dependent on contributor-scan. Groups 1–7 and tasks 8.1, 8.2, 8.4 are verified; 26/27 executable tasks complete. Remaining release work follows [M1–M5](../specs/pre-push-safety-net/release-milestones.md). This amendment creates no duplicate implementation spec and resets no prior approvals. @@ -289,17 +291,16 @@ supersede this accepted sequence. Blocking deployment follows a later explicit d The sections below preserve prior plans and measurements. Statements about the current bottleneck, PHP coverage, rollout order or feature readiness are superseded by the current section above. -**Rewritten 2026-09-09.** The previous version (kept as `.kiro/archive/steering/roadmap-2026-09-04.md`, outside steering) described a pre-Joern world — +**Rewritten 2026-09-09.** The previous version (kept as `.kiro/archive/steering/roadmap-2026-09-04.md`, outside steering) described a pre-Joern world, "adjudication is still Python-`ast` plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was -adopted, the model layer was built, wired and measured, and two repositories have been scanned. See +adopted. The model layer was built and wired, then measured. Two repositories have been scanned. See `overview.md` for what is true today; this is what happens next and in what order. ## The target: a first release for small projects > "a first release that is ready for smaller projects like libpng or wordpress" -Those two are, today, exactly the cases that do not work — and they fail for **different** reasons, which is -what makes them a good pair of targets rather than one. +Neither works today. They fail for different reasons, so both are useful targets. | target | why it fails now | what a release needs | | --- | --- | --- | @@ -312,8 +313,8 @@ So "ready for" has to mean two different things, and the release criteria say wh * For one it **cannot** (libpng today): run inside its envelope, report nothing false, and *state the limit* rather than returning a quiet clean bill of health. -The second is not a consolation. A tool that says "I do not model this class" is usable; one that silently -finds nothing is the thing that teaches developers to distrust SAST. +A tool that says "I do not model this class" is usable. Silent failure to find anything teaches developers to +distrust SAST. ### Release criteria (v0.1) @@ -321,15 +322,15 @@ finds nothing is the thing that teaches developers to distrust SAST. 2. **Zero known-false entailed findings** on every pinned checkout. 3. Finds the known vulnerability wherever a family can decide it in principle (Req 4.6). 4. States what each family cannot decide, in the report (Req 5.6). -5. The report is proportionate — no rule repeating itself past a stated count (Req 5.5). +5. The report is proportionate: no rule repeats itself past a stated count (Req 5.5). 6. Installs and runs without the user having Joern, a JVM, or a PHP interpreter (docker compose + wrapper). 7. A committed regression baseline per target, so the next change is measurable rather than argued about. ### What that reorders -PHP moves **up**: it is the shortest path from here to a target the maintainer named, and it is the case the +PHP moves up: it is the shortest path from here to a target the maintainer named, and it is the case the existing arbiter is best suited to. The C decision moves up with it, because criterion 4 needs `c/memory` -either earning its place or being scoped — and scoping it is cheap. +either earning its place or being scoped, and scoping it is cheap. ## The one test everything is ordered by @@ -337,40 +338,40 @@ either earning its place or being scoped — and scoping it is cheap. > capabilities but never use it" So the ordering principle is: **make the output worth reading, then make it find more.** A better arbiter -behind an unreadable report changes nothing, and a second language before the report is fixed just doubles the +behind an unreadable report changes nothing, and a second language before the report is fixed doubles the noise. ## Now — make a repository scan finish at all -The output work this section used to hold is done, and PHP went considerably further than it planned: three -CVEs on real WordPress plugins, entailed by the graph with no model in the loop, each dropping on its fixed -side. The detector is not the problem any more. **Finishing is.** +The output work previously listed here is done. PHP detection exceeded the plan: the graph entailed three +CVEs on real WordPress plugins without a model, and each finding disappeared on the fixed side. The remaining +problem is getting scans to finish. A 637-file plugin builds in 76 seconds and then the taint query does not complete. Everything below is ordered by one question: what is the shortest path to a scan of a real plugin that reports its CVE? -1. **5.14 — stop asking questions that cannot have an answer.** 53% of taint requests are put to a family +1. For **5.14**, stop asking questions that cannot have an answer. 53% of taint requests are put to a family whose sinks do not appear in scope; every one of PMPro's 487 `deserialization` requests is asked of a - repository that never calls `unserialize`. Arithmetic, not judgement, and it cannot lose a finding. + repository that never calls `unserialize`. This is an arithmetic check and cannot lose a finding. 4.6 hours → 2.2. Do this first because it makes every later measurement cheaper to run. -2. **5.12 — rank, because 84% of regions currently share one rank ordered alphabetically.** Two thirds of the - budget is spent arbitrarily. Fifty well-chosen regions beat five hundred; combined with (1) that is the +2. For **5.12**, add ranking. Currently 84% of regions share one rank and are ordered alphabetically. Two thirds of the + budget is spent arbitrarily. Fifty well-chosen regions are better than five hundred; combined with (1) that is the difference between hours and minutes. The ranker is an LLM job, and it is safe as one because ranking - cannot manufacture a finding -- it decides what is looked at, never what is reported. - **The loop is closed and that is a design requirement, not a caveat:** a region the ranker excludes is + cannot manufacture a finding. It decides what is looked at, never what is reported. + The loop is closed and that is a design requirement, not a caveat: a region the ranker excludes is never arbitrated, so it never produces a label, so the ranker never learns it was wrong. An exploration slice and a small fully-labelled corpus are what make the signal unbiased. -3. **5.13 — the 6.63 s/request itself**, if (1) and (2) are not enough. `callDepth=3` over 7,061 methods is - the term that makes a request cost seconds; the two-stage joins are NOT the cost (measured: 22%, and they +3. For **5.13**, reduce the 6.63 s/request itself if (1) and (2) are not enough. `callDepth=3` over 7,061 methods is + the term that makes a request cost seconds; the two-stage joins are not the cost (measured: 22%, and they found two of the three CVEs). Only after a scan finishes is there any point tuning what it finds. ## Next — the families that cannot be asked -4. **5.9 — PHP obligation facts.** `dominance_specs(language="php")` is empty, so missing-capability checks +4. For **5.9**, add PHP obligation facts. `dominance_specs(language="php")` is empty, so missing-capability checks are not asked on any WordPress scan. Not a miss; nothing was measured. -5. **5.6 — a sanitizer cleanses the family it cleanses.** The list is flat, so `esc_sql` reads as sufficient +5. For **5.6**, associate each sanitizer with the family it cleanses. The list is flat, so `esc_sql` reads as sufficient for XSS and `esc_html` as insufficient for SQL. Wrong in both directions, so no `output_encoding` verdict currently means anything. 6. **File the php2cpg defect upstream.** A `global` inside a closure yields a CPG Joern's own dataflow @@ -379,12 +380,12 @@ Only after a scan finishes is there any point tuning what it finds. ## Then — close the deferred engine questions -8. **2.14 — control dependence.** Absent from both CPGs, so every guard question runs on CFG dominance. Find +8. For **2.14**, check control dependence. It is absent from both CPGs, so every guard question runs on CFG dominance. Find out whether Joern's CDG pass can be applied from the query script and what it costs. If it cannot, that is a real bound on what this engine can reason about and should be recorded as one. -9. **2.13 — a destination allocated to size.** Deliberately blocked on 6, so the rule can be measured for what +9. **2.13** covers a destination allocated to size. It is deliberately blocked on 6, so the rule can be measured for what it *silences* as well as what it clears. -10. **Group 3–4 — ship it.** Contributor output, the Docker image, the memory-family decision, retiring +10. Ship **Group 3–4**: contributor output, the Docker image, the memory-family decision, and retirement of `evidence_level` now that the rung carries the meaning. ## Alongside — keep the codebase from growing into the problem @@ -393,16 +394,16 @@ Only after a scan finishes is there any point tuning what it finds. `.kiro/specs/ir-consolidation/` (brief, unapproved) separates two questions with opposite answers: -* **The source IR is already carried, not about to be added.** `semantic/` is 3,218 lines — 16% of the - codebase — holding a hand-written IR and flat taint engine measured at **2.2% entailment against the CPG's +* The source IR already exists in the project. `semantic/` is 3,218 lines, or 16% of the + codebase. It holds a hand-written IR and flat taint engine measured at **2.2% entailment against the CPG's 41.3%**, still running as a proposer. Vine and Ghidra P-Code are *binary* IRs and do not help here; the - work is subtraction. A candidate ~1,000-line deletion with a number attached. + work is subtraction. This is a candidate ~1,000-line deletion supported by a measurement. * **The missing abstraction is C arithmetic**, and there an existing IR is exactly right. GhidraPAL's three-valued-logic abstract interpreter over P-Code decides "can this index exceed this allocation", which is the class `c/memory` cannot see and every libpng overflow is. Vine is dormant since 2009 and is not the candidate. -The ordering constraint is the point: **question 1 lands before question 2 starts**, or adopting an IR makes +The order matters: **question 1 lands before question 2 starts**, or adopting an IR makes the growth problem worse rather than better. ## After — the feedback loop @@ -413,15 +414,15 @@ the growth problem worse rather than better. ## Not now, and why -- **`authorization-obligations`** (24 open tasks, approved) — the dominance arbiter delivered its substance by +- `authorization-obligations` (24 open tasks, approved): the dominance arbiter delivered its substance by another route. It should be reconciled or retired rather than worked, and carrying it as "approved, not started" misrepresents both. -- **`reachability-flow-model`** (23 open, nothing approved) — largely delivered by adopting Joern. Same +- `reachability-flow-model` (23 open, nothing approved) was largely delivered by adopting Joern. Same treatment. -- **`constrained-detector`** (4 of 20, tasks unapproved) — parked. -- **More corpus volume.** 198 of 238 OWASP pairs were left unvendored deliberately: volume without +- `constrained-detector` (4 of 20, tasks unapproved) is parked. +- More corpus volume is deferred. 198 of 238 OWASP pairs were left unvendored deliberately: volume without readability makes every number worse, and the corpus is not the binding constraint. Reachability is. -- **A second CPG engine, a server mode, an execution rung.** Server mode was measured and declined (6% of a +- A second CPG engine and an execution rung are deferred, as is server mode. Server mode was measured and rejected (6% of a scan). The other two have no evidence calling for them yet. ## How to know it is working @@ -430,6 +431,6 @@ The scoreboard in `overview.md`, and one question: **can a developer run this on write and get something they act on?** Today that is true for one Python application and false for one C library, and the specs above are ordered by what changes that. -A release is ready when that question is answered **yes for WordPress and honestly for libpng** — found for -the one whose class we decide, and a stated limit for the one we do not. Both are release criteria; only one +A release is ready when it answers **yes for WordPress and honestly for libpng**. It must find the vulnerability +where the engine can decide the class, and state the limit where it cannot. Both are release criteria; only one of them is a detection claim. diff --git a/README.md b/README.md index dbf4c51..0ceb7d2 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # OpenUltraSAST -OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on -repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It -keeps what it can prove apart from what it only suspects. +OpenUltraSAST is an experimental static security analyser. It aims to check security on +repositories it has never seen. It scans a checkout or the commits a `git push` would publish. It +distinguishes what it can prove from what it suspects. A scan combines up to four parts: @@ -11,18 +11,18 @@ A scan combines up to four parts: - a Joern code property graph (a graph of the program's syntax, control flow and data flow); - optionally, a model. It is asked only the question the graph cannot settle. -Every finding carries the evidence that earned its place. When a scan could not look at -something, it records a degradation. It does not report a clean result. +Every finding includes its supporting evidence. When a scan cannot check something, it records +a degradation instead of reporting a clean result. Where a repository's application can be built and run, an experimental proof arm goes further. It -demonstrates a finding by executing it: the issue triggers on the vulnerable revision and not on -the fixed one, so a demonstrated finding is a reproducible fact, not a score. This arm is in pilot -and being qualified on repositories the tool has never seen; see [Status](#status). +demonstrates a finding by executing it. The issue triggers on the vulnerable revision and not on +the fixed one, which makes the finding reproducible. This arm is in pilot testing and is being +qualified on repositories the tool has never seen; see [Status](#status). `ousast` is the interface. Model work over whole repositories runs on a separate agentic plane (google/ax). A learned decision engine is in development. A scan needs neither of them. -This file is the front door. Each section links to its page of the documentation site. +Each section in this file links to the corresponding page on the documentation site. The site is published at . Its sources are in `docs/` and `mkdocs.yml`; to preview it locally, run `uv sync --extra docs && uv run mkdocs serve`. @@ -30,24 +30,24 @@ The site is published at . Its source **v2.0.1, state as of 2026-10-08.** The pre-push safety net is **not yet released**. Blocking on the static analyser by hand is **NO-GO** against every M4 gate, and the project no longer tries to -reach them that way. +meet those gates that way. - No blocking hook capability is enabled; the pre-push hook is advisory only. - Two independent populations (case sets frozen before any scan) have been spent on the static - engine: recall was 1 of 11, then 0 of 17. The learned decision engine ranks but blocks for no + engine: recall was 1 of 11, then 0 of 17. The learned decision engine ranks findings but does not block for any family. -- Why blocking is hard: static abstract interpretation could not prove the precision a block - needs. So the current work is a **dynamic proof arm**. It builds and runs a repository's app and +- Static abstract interpretation could not prove the precision needed to block a push. Current + work focuses on a **dynamic proof arm**. It builds and runs a repository's app and demonstrates a finding by a differential: the issue triggers on the vulnerable revision and not - on the fixed one. A demonstrated finding is ground truth by execution, so it can earn a block + on the fixed one. Execution establishes the finding as ground truth, so it can justify a block without calibration. Pilot demonstrations exist, including one executed on the cluster; the arm is being qualified, not shipped. - The release gate is a **measured** catch-rate and false-alarm number on repositories the tool - has never seen (the unseen-repo evaluation), produced rather than asserted. Until that number + has never seen (the unseen-repo evaluation). Until that number exists, treat all output as advisory. -Every detection change has to win a controlled measurement on held-out repositories. One page -states where things stand, with the record behind each figure: +Every detection change must show an improvement in a controlled measurement on held-out repositories. +The current status and the record behind each figure are in [docs/where-we-stand.md](docs/where-we-stand.md). For developers, the pre-push check and its honest limits: [docs/pre-push-for-developers.md](docs/pre-push-for-developers.md); the catch–prove–fix–re-verify loop: [docs/use-case-fix-and-reverify.md](docs/use-case-fix-and-reverify.md). @@ -85,7 +85,7 @@ cd /path/to/your/repository && ousast pre-push install # remove: ousast pre-p What it cannot do: -- There is no engine. Nothing follows a value across statements or files. +- There is no engine. It cannot follow a value across statements or files. - The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower bounds run from 0.03 to 1.0 on the real pins (`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`). @@ -95,9 +95,9 @@ the overlay. ### Tier 2: standard and deep scans (Docker) -`standard` and `deep` need Joern. The Docker image ships three things in one image: the tool, -Joern v4.0.625 and `php-cli` (the PHP frontend needs it). Each release publishes it to GHCR. It is -big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the +`standard` and `deep` need Joern. The Docker image includes the tool and +Joern v4.0.625. It also includes `php-cli`, which the PHP frontend needs. Each release publishes the image +to GHCR. It is big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the shell wrapper uses: ```bash @@ -131,7 +131,7 @@ Details, including the host install, are in [ops/README.md](ops/README.md). Keys and store settings live in `.env` in the working directory. The file is gitignored; `.env.example` lists the names. -- `DEEPSEEK_API_KEY` serves every LLM call. +- Every LLM call uses `DEEPSEEK_API_KEY`. - `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`). - `.env` never overrides a variable already exported in your shell. If a key change seems to have no effect, `unset` the stale export. @@ -141,7 +141,7 @@ Keys and store settings live in `.env` in the working directory. The file is git ### Tier 3: the plane on ax (maintainers) -Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's +Tier 1 and 2 need none of this. The current profile is a single-node kind cluster on the maintainer's host. - `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to @@ -153,7 +153,7 @@ host. - The verify and roles tasks need `DEEPSEEK_API_KEY`. - Memory lives in the local file store or the S3 store. RustFS is the tested server. -Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md). +See [ops/ax/README.md](ops/ax/README.md) for bring-up and doctor instructions, along with the smoke Run. ### Tier 4: Kubernetes (k3s), planned @@ -166,7 +166,7 @@ submit a Run in a remote execution mode. It needs: - runner and engine images published to GHCR and pinned by digest; - the S3 store as the artifact path, with no receiver on a laptop. -The plan, and the code changes it needs with file paths, are in +The plan and required code changes, with file paths, are in [docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster). ## Scan modes @@ -209,9 +209,9 @@ The tool uses three representations of code, from cheap to expensive: 3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and files. -Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that -should have been guarded. They are found as obligations, dominance and configuration census. A -model answers only the residual question. +Taint analysis uses fact tables and queries on the graph to produce verdicts. Absence bugs involve +operations that should have been guarded. The tool finds them through obligations and dominance +checks, plus a configuration census. A model answers only the residual question. - [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the code it comes from and what it misses. @@ -229,7 +229,7 @@ The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% break a known detection. They are not a recall estimate on real code, and PHP is not in the gate. Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in -`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off. +`src/openultrasast/ruleset/frameworks.toml`. Its contribution can be measured and switched off. ## The plane on ax @@ -282,8 +282,7 @@ refuses to run and names each missing piece. It does not fall back. ## Decision engine and evaluation -A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It -follows three rules: +A learned decision engine is being built to replace hand-tuned detection edits. Its rules are: - Every instrument produces signals, never verdicts. - Labels come from ground truth only. @@ -301,7 +300,7 @@ It was measured on six families, on repositories the compiled program was not sh (`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`). Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`. -Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md). +The design and evaluation slices are in [docs/decision-engine.md](docs/decision-engine.md), along with the A/B protocol. [docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers @@ -392,15 +391,15 @@ settings so the actors can pull it without registry credentials. The task image contains the current engine and worker code. Rebuild it after code changes and pass its digest with `--image`; no analyzer download is needed. The host -materializes each pin as usual and uploads source and prepared questions under +prepares each pin as usual and uploads source and prepared questions under `engine-queue///`. Only object-scoped presigned URLs and deadline parameters enter Task env. Credentials stay on the coordinator. Configure `OUSAST_MEMORY=s3://…` and the store credentials in the coordinator's `.env` or environment, with `S3_ENDPOINT=https://files.because-security.com`. Presigned links expire after the pin deadline plus ten minutes. IP and HTTP endpoints are rejected: the operator's -actor EgressPolicy must allow **files.because-security.com:443**. +actor EgressPolicy must allow files.because-security.com:443. -Completion is the result object, even while AX reports Running. Failed phases and +The result object marks completion, even while AX reports Running. Failed phases and host deadlines produce explicit failure/timeout rows. Task and staging-object cleanup runs on success and failure; cleanup failure stops further dispatch. Per-pin files, `--rerun-status`, resume and `progress.json` retain the existing trace-runner behavior. @@ -440,7 +439,7 @@ uncommitted changes without publishing a git commit. With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in `.env`, run a one-pin smoke using the default experiment units and local example memory (or add `--examples /path/to/frozen-examples.jsonl`). Use the `ax` binary at -`$HOME/go/bin/ax` — `$HOME/.local/ax-v0.3.1/ax` hits the AX-lane dispatch blocker: +`$HOME/go/bin/ax`. `$HOME/.local/ax-v0.3.1/ax` hits the AX-lane dispatch blocker: ```bash : "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"