marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: retire the SZZ-pool metric, add the valid-benchmark brief, plain-English pass

Browse Source 
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in
the wrong place about half the time, and ~76% of entries are library-internal (not
reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise.
Retired as a metric and a gate across the roadmap and briefs; no pool-scored number
is quoted anywhere.

Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the
fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified;
ordinary negatives from the same repos; unseen repos; the whole tool measured; plain
metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch).

Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two
active briefs via Codex, then reviewed the diff: only these files changed, every
number/path/hash/command is unchanged, and the retired-metric decision is intact.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-12 16:38:18 +02:00
co-authored by Claude Opus 4.8
parent c3f610f9f4
commit b08f372022
4 changed files with 455 additions and 371 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/engine-change-scoped-analysis/brief.md
+212 -213
View File
@@ -1,93 +1,93 @@
# Engine change-scoped analysis (G2 workstream)
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks a concrete lever for the
engine-detection gap found while driving the G0 baseline. This is G2 ("improve what moves the unseen-repo
numbers"), gated on the pool metric; it is NOT part of the G0 record-07 baseline (that runs on the quick-rule tier).
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks work to address the
engine-detection gap found during the G0 baseline run. This is G2 ("improve what moves the unseen-repo
numbers"), gated on the pool metric; it is not part of the G0 record-07 baseline (that runs on the quick-rule tier).
## Problem (measured, not assumed)
On real unseen vulnerability changes the engine produces **no finding** — but the diagnosis (live, 300 s long-deadline
replays) shows it is usually **not a detection-capability gap**:
The engine produces **no finding** on real unseen vulnerability changes. Live 300 s long-deadline replays show
that the cause is usually not a detection-capability gap:
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
`change_context_incomplete`. **Zero analyses ran.** And the ranked candidates were **repo-wide and unrelated to
`change_context_incomplete`. No analyses ran. The ranked candidates were **repo-wide and unrelated to
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
re-audit the whole repo and runs out of budget before analyzing the change.
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
`contributor-scan ... projection_unavailable`. Multi-causal; correspondence was one part (fixed separately,
commit c39fbb8), capability registry is Task 8.3.
`contributor-scan ... projection_unavailable`. This has several causes. Correspondence was one (fixed separately,
commit c39fbb8); the capability registry is Task 8.3.
## Hypothesis
A pre-push hook should vet **the code this push changed**, not re-scan the whole repository. Seeding/scoping the
engine's analysis to the **changed declarations and their dataflow neighborhood** (the `change_sites` are already in
the replay spec) should let analysis **actually run within budget** instead of being deferred — which is where
detection is being lost, not in Joern's capability.
A pre-push hook should check **the code this push changed**. Starting the engine's analysis from the
**changed declarations and their dataflow neighborhood** should let it run within budget instead of being
deferred. The `change_sites` are already in the replay spec. The missed detection comes from deferral,
rather than a limit in Joern's capability.
## Anchors (confirm in increment 1)
- `src/openultrasast/rank.py` — the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
- `src/openultrasast/mapping.py` — builds the `selected`/`deferred` scope + the budget allocation that defers on
- `src/openultrasast/rank.py` is the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
- `src/openultrasast/mapping.py` builds the `selected`/`deferred` scope + the budget allocation that defers on
`deadline_exhausted`.
- `src/openultrasast/push/policy.py`, `push/runner.py` — consume the scope; change_sites arrive via the push input.
- `benchmarks/unseen/replay_entry.py` — carries `changed_paths` / `declaration_paths` in the spec.
- `src/openultrasast/push/policy.py` and `push/runner.py` consume the scope; change_sites arrive via the push input.
- `benchmarks/unseen/replay_entry.py` carries `changed_paths` / `declaration_paths` in the spec.
## Increments (progress tracking)
- [x] **0. Cost split confirmed (2026-10-11, timed live replay).** `build_seconds=66.69` (CPG build is cheap),
`query_seconds=0.0` (no query ever executed), total 303 s (hit the deadline), **4,095** candidate questions all
`analysis_not_started=deadline_exhausted`, `selected: []`, `regions_scanned=0`. So of the 303 s, ~67 s built the
CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change —
the budget was spent before a single query ran. The CPG build is NOT the bottleneck; the over-broad candidate set
is. Change-scoping is confirmed as the right lever, and because per-candidate prep includes the line-correspondence
work, fewer candidates also cuts that cost (the two fixes compound).
CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change.
The budget was spent before a single query ran. The over-broad candidate set is the bottleneck, rather than the CPG build.
This confirms change-scoping as the fix. Preparing each candidate includes line-correspondence work,
so fewer candidates also reduce that cost.
- [x] **1. Located (2026-10-11).** The over-enumeration is upstream of the scan, not inside it:
- `push/runner.py:345-348` — `head_regions = discovery(tip.root, …)` discovers regions across the **whole head
- `push/runner.py:345-348`, `head_regions = discovery(tip.root, …)` discovers regions across the **whole head
tree** (the ~4,095), independent of the change.
- `push/runner.py:397-407` — those regions are passed wholesale to `scan_repository(..., change_context=context)`;
- `push/runner.py:397-407`, those regions are passed wholesale to `scan_repository(..., change_context=context)`;
`change_context` is threaded in but **never scopes the region set**.
- `model/scan.py:167` and `:364` — `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all
- `model/scan.py:167` and `:364`, `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all
regions × families; with `order_by_evidence=True` (`runner.py:288/403`) the scan evidence-tiers **every** pair
before analysis — that pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`.
before analysis. That pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`.
- Primitive already present: `model/regions.py:279` computes the changed-path set from the context;
`compare_targeted_base` (`runner.py:411`) is a precedent for change-targeted work.
- **Fix locus (increment 2):** restrict the *analyzed* region set to the change's declarations + dataflow
- For increment 2, restrict the *analyzed* region set to the change's declarations + dataflow
neighbourhood (filter `head_regions` by `context.changed_paths` before the scan, or scope the enumeration inside
`scan_repository`). Keep the **CPG whole-repo** so taint queries can still trace sources from unchanged code into
the changed regions — only the question set is change-scoped.
the changed regions. Only the question set is change-scoped.
**Correction (2026-10-11), after a reverted first attempt — two things the naive filter got wrong:**
Correction (2026-10-11), after reverting the first attempt: the filter got two things wrong.
- **The engine already reaches into unchanged code by design, and it is tested.** A changed `source` that reaches
an unchanged `sink` must still select the unchanged sink's question
(`tests/test_model_change_context.py::test_changed_dependency_reaches_unchanged_ranked_question`;
`test_change_does_not_replace_ranker_selection` pins that change context must NOT alter the ranker's selection).
`_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths` — which locations to itemise vs
summarise — NOT to shrink the tier population. A "changed files only" scope deletes the reach-into-unchanged
`test_change_does_not_replace_ranker_selection` pins that change context must not alter the ranker's selection).
`_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths`. This controls which locations to itemise vs
summarise, rather than shrinking the tier population. A "changed files only" scope deletes the reach-into-unchanged
behaviour. Any scope must keep changed regions **plus the regions they reach** (the affected/`relationships`
neighbourhood), not changed files alone.
- **The 236 s is now split (2026-10-11, saved artifacts).** `total=291s build=52s query=0s arbitrate=238.75s`,
and `query_seconds_by_kind["evidence"]=238.54s` — the evidence-tier pass is **99.9 %** of the arbitrate phase;
and `query_seconds_by_kind["evidence"]=238.54s`, the evidence-tier pass is **99.9 %** of the arbitrate phase;
`affected_context` + `_scope_work` are **0.21 s**. The pass tiered 1,015 pairs (`tier_counts 0:1009, 1:6`) at
**~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget — `selected=0`,
**~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget, `selected=0`,
`deferred=1365` all `deadline_exhausted`. Secondary anomaly: the design assumes ~2.5 ms/pair
("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is pathologically slow on a cold large
CPG — a separate optimisation from cutting the pair count.
- **So increment 2's single target is the evidence pass** (`scan.py:338-359`, `_evidence_pass`). The fork is HOW
to cut it without deleting reach-into-unchanged:
- (i) **Scope + bound:** order the evidence candidates change-first and bound the pass to a fraction of the
("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is unusually slow on a cold large
CPG. Fixing that is separate from cutting the pair count.
- Increment 2 targets the evidence pass (`scan.py:338-359`, `_evidence_pass`). The options must
reduce its cost while preserving flows into unchanged code:
- (i) Scope + bound: order the evidence candidates change-first and bound the pass to a fraction of the
budget so analysis queries still run; reached unchanged regions are tiered only while budget lasts.
- (ii) **Neighbourhood pass:** tier the changed regions, read the reached regions from the context rows the
pass already returns, tier those in a second small pass — exact reach-into-unchanged, more code.
- (iii) **Per-pair speedup:** attack the ~235 ms/pair (warm server / batching) rather than the pair count —
helps every scan but is a deeper engine change and does not by itself bound a huge repo.
- (ii) Neighbourhood pass: tier the changed regions, read the reached regions from the context rows the
pass already returns, then tier those in a second small pass. This preserves exact reach-into-unchanged behaviour but needs more code.
- (iii) Per-pair speedup: reduce the ~235 ms/pair (warm server / batching) rather than the pair count.
This helps every scan but needs a deeper engine change and does not by itself bound work on a huge repo.
**CORRECTION (2026-10-11, after implementing (i) + four live re-verifies — the cost model was wrong).**
**CORRECTION (2026-10-11, after implementing (i) + four live re-verifies, the cost model was wrong).**
Scope + bound was built and tested (change-first ordering, adaptive budget-fitted portioning, `_context_pass`
scoped to changed files, honest `evidence_pass_bounded` degradation; branch `engine-change-scoped-evidence-pass`).
It correctly cut the tiered pair count — but `query_seconds` stayed **0** every time. The live numbers show why:
It cut the tiered pair count, but `query_seconds` stayed **0** every time. The live numbers show why:
| pairs tiered | run_batch calls | evidence s | build s |
|---|---|---|---|
@@ -99,53 +99,53 @@ detection is being lost, not in Joern's capability.
Evidence time is **flat regardless of pair count** → the cost is **per-*call* fixed** (~150–220 s: loading the CPG
into the query engine + the census the evidenceOnly batch computes), ~85 ms/pair on top. Pair-count scoping
therefore cannot free budget, and portioning re-pays the fixed cost per call (so (i) can be counterproductive).
Build is also large and swings 52→127 s — two engines thrashing the 7 GB host (the record-07 baseline shares it).
The real levers are now (iii)-shaped, not (i): **warm/persistent query engine or a cached census** to kill the
~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s; plus a **clean (uncontended)
measurement**, because contention alone may be inflating build + fixed cost 2–3×. Scope+bound is NOT merged.
Build time also varies from 52→127 s. Two engines compete for the 7 GB host, which the record-07 baseline shares.
The fixes now resemble (iii), rather than (i): a **warm/persistent query engine or a cached census** to remove the
~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s. A **clean (uncontended)
measurement** is also needed because contention may inflate build + fixed cost 2–3×. Scope+bound is not merged.
**UNCONTENDED measurement (2026-10-11, baseline paused via SIGSTOP + container kill, then resumed).** Shipped
engine, one evidence call over 1,015 pairs: `build=48 s`, `evidence=243.79 s`, `query=0`. The evidence pass is
**the same ~244 s uncontended as ~238 s contended** → the cost is **INTRINSIC, not contention**; contention only
**the same ~244 s uncontended as ~238 s contended** → the cost is **intrinsic, not contention**; contention only
inflated `build` (48 → 127 s) and added run-to-run noise. The model is `evidence ≈ 150 s fixed (CPG load into the
query engine + the census the evidenceOnly batch computes) + ~93 ms/pair` (244 = 150 + 1015×0.093). Consequences:
- **~200 s of fixed overhead** (`build` + evidence-load) precedes any analysis, so a 300 s budget leaves only
~100 s for everything else. This is why `query=0` even on a clean host.
- **Change-scoping still helps, but only by cutting the per-pair tail** (~1,015×93 ms ≈ 95 s): one evidence call
over ~30 changed pairs is ~150 s + 3 s instead of 244 s, which would free ~90 s — possibly enough for analysis,
but marginal.
over ~30 changed pairs is ~150 s + 3 s instead of 244 s. This would free ~90 s, which might leave enough time
for analysis, but only just.
- **It must be ONE call, not portioned.** The ~150 s is a first-call load, so the adaptive probe attributed it to
its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4 — the implemented
scope+bound is doubly wrong for this cost model. The correct shape: change-first candidates, count-capped, ONE
its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4, the implemented
scope+bound has both problems under this cost model. The correct shape: change-first candidates, count-capped, ONE
evidence call, `deadline=None`.
- **Pre-push viability:** the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a
- Pre-push viability: the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a
repo this size without a warm/persistent session or a cached census. A larger budget (e.g. 600 s) makes the
benchmark arm run but is not a realistic pre-push deadline. The lever that matters for the product is killing
the ~150 s fixed cost, not scoping pairs.
- [ ] **2. Seed analysis from the change.** Prioritize (or restrict first-pass to) the changed declarations plus
their call/data neighborhood over repo-wide static rank, within budget.
- [ ] **3. Soundness.** Keep cross-file dataflow INTO the changed function (don't scope to literal changed lines);
- [ ] **3. Soundness.** Keep cross-file dataflow into the changed function (don't scope to literal changed lines);
add tests; keep existing push tests green.
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now RUN and produce a finding on
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now run and produce a finding on
`install_packages` (vs `deferred`)? Re-run metrics-plugin.
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate) — needs the
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate), needs the
kube lane (blocked on the AX completion path) for scale.
## Dependencies / notes
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; the win
here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
- Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine
changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here.
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; this work
moves findings from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
- Implement directly and verify on the live instrument (a fake-client/unit pass is not evidence the live engine
changed); the kube-lane and store endpoints are infrastructure owned by the operator.
## Efficiency barrier analysis (post-record-07 priority)
**What the record-07 baseline actually measured (verified on the records): regex only.** The default and
long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0` — **no LLM is
called** — and the engine never completed (`engine_covered=0`, the ~200 s wall). So the measured `catch≈2.5%` is
the **quick pattern-rule tier alone**: essentially grep-with-structure. This is the FLOOR, not the product, and it
confirms the maintainer's point — pattern rules alone will never be enough, or `grep` would do. The product's
detection lives in two tiers that did NOT run:
The records confirm that the record-07 baseline measured regex only. The default and
long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0`. No LLM is
called, and the engine never completed (`engine_covered=0`, the ~200 s wall). The measured `catch≈2.5%` covers
only the quick pattern-rule tier, a structured pattern search. This measures the floor, rather than the product.
It confirms the maintainer's point that pattern rules alone are insufficient; otherwise `grep` would do.
The product's two detection tiers did not run:
| Tier | Mechanism | Measured cost (uncontended, mid-size repo) | Ran? | Catch |
|---|---|---|---|---|
@@ -153,20 +153,20 @@ detection lives in two tiers that did NOT run:
| 2 engine (Joern) | CPG build + taint/dataflow | build 48 s + evidence (150 s fixed + 93 ms/pair) + analysis (unmeasured) | no (walled) | unknown |
| 3 LLM adjudication | model judgment / search-with-proof (hunter, fusion, decision engine) | model latency + $/call (unmeasured in the push path) | no (disabled) | unknown |
**The barrier:** a releasable net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window —
target **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis is a budget ledger per tier against
that window, and the measurements each needs:
To be releasable, the net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window.
The target is **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis records each tier's budget against
that window and identifies the measurements needed:
- **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the decisive architectural
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the architectural
test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size.
Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail.
- **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall LIFT of tier 3 over tiers
1+2 on the pool — does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
- **Deliverable:** a **barrier table** — for budget ∈ {60, 120, 300, 600} s × each tier combination, does it FIT
and what is the MEASURED recall/false-alarm on the pool — identifying the minimal viable config for a releasable
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall lift of tier 3 over tiers
1+2 on the pool, does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
- Produce a **barrier table** for budget ∈ {60, 120, 300, 600} s × each tier combination. Record whether it fits
and its measured recall/false-alarm on the pool. Use this to identify the minimal viable config for a releasable
net. 600 s unblocks the benchmark arm but is not a pre-push deadline; the product target is tiers 2+3 under ~120 s.
Sequence: finish G0 (record 07) → **E1/E2 first** (they decide whether the warm session alone crosses the engine
@@ -176,8 +176,8 @@ lever.
## E1/E2 RESULTS (measured 2026-10-12, CLEAN host, bedrock-agentcore, 3-file change, 300 s budget, head side)
Four clean-host replays settle the engine barrier and **correct the earlier "~200 s graph-load wall"**, which was
conflated with host contention from the running baseline replay:
Four clean-host replays **correct the earlier "~200 s graph-load wall"** diagnosis. It had confused graph-load
cost with host contention from the running baseline replay:
| Run | build | evidence (taint batch) | dominance | query_s | selected | deferred |
|---|---|---|---|---|---|---|
@@ -186,40 +186,40 @@ conflated with host contention from the running baseline replay:
| warm session `SESSION=1` (clean) | 32.9 | 228.6 | 25.9 | 25.9 | **356** | 1009 |
| session + graph preload (clean) | 30.4 | 223.2 | 32.5 | 32.5 | **356** | 1009 |
**Findings (each traced to a measured cause, not assumed):**
The measurements show:
1. **The engine arm was walled by CONTENTION, not an intrinsic floor.** The prior `query=0 / selected=0 / all
1. Host contention prevented the engine arm from running. The prior `query=0 / selected=0 / all
1365 deferred` was measured while the slice-1 baseline replay contended for the 7 GB host. On a clean host the
**shipped (disposable) engine already runs dataflow**: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**,
shipped (disposable) engine already runs dataflow: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**,
dominance flow queries complete. This is the first time the engine arm produced analysis on this change.
2. **The evidence/taint batch (~225 s) is INVARIANT to session and to graph preload** (225.1 / 228.6 / 223.2 across
2. **The evidence/taint batch (~225 s) is invariant to session and to graph preload** (225.1 / 228.6 / 223.2 across
the three clean runs). Warm session and preload-at-build do **not** reduce it. So the dominant cost is intrinsic
to the taint batch, not graph I/O.
3. **Graph-load is cheap — it is NOT the wall.** The `dominance` query is a full cold query *including*
3. Graph-load is cheap and does not account for the delay. The `dominance` query is a full cold query *including*
`importCpg`, and it costs only ~35 s. So `importCpg` is tens of seconds, not ~150 s. **This corrects the earlier
note** that attributed the ~150 s fixed to "first query JVM-start + graph-load + census." The ~160 s fixed lives
inside `taint.sc`'s own per-batch flow work (seed/sink memo fill over the distinct `(family, file)` combinations
it walks), paid lazily as the first pairs process — independent of the raw pair count.
it walks). It is paid lazily as the first pairs are processed, independent of the raw pair count.
4. **The warm session is not the lever.** It starts fine on a clean host (the contended `session_startup_timeout`
was contention-only), but gives ~no benefit here: subsequent queries (dominance) are already cheap disposable,
and the evidence pass is a single batch call, so there is nothing for the session to amortise.
5. **The barrier, restated:** `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer
as `deadline_exhausted`; only 356 are analysed. The lever is **reducing the taint batch's work** — change-scope
the pair/family set so the batch fits budget — NOT session, preload, or a bigger JVM. The parked branch
5. The barrier, restated: `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer
as `deadline_exhausted`; only 356 are analysed. The fix is to **reduce the taint batch's work** by scoping
the pair/family set to the change so the batch fits the budget. Session reuse, preload, or a bigger JVM will not do this. The parked branch
(`engine-change-scoped-evidence-pass`) is therefore **relevant after all, on a clean host**, provided it scopes
the family/file set `taint.sc` walks (not merely the pair list); its adaptive portioning still needs replacing
with a fixed cap (per-call-fixed cost, not per-pair).
6. **Side note:** the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run — a separate
6. Side note: the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run, a separate
intermittent base-revision build issue; the `head` side built and analysed normally. Track separately.
**What E1/E2 did NOT answer (the real G2 question):** on a clean host the engine selects 356 regions — *does it
CATCH vulns regex misses?* That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the
engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the decisive
E1/E2 did not answer the G2 question: does the engine catch vulnerabilities regex misses when it selects 356 regions
on a clean host? That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the
engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the
next experiment, ahead of E3/E4/L1/L2.
## ENGINE-ON PILOT RESULTS (measured 2026-10-12, CLEAN host, 5 public pool repos, vuln change each, 300 s)
Five public repos picked for flow-shaped families and small size (engine's best chance to finish and catch):
Five public repos were picked for flow-shaped families and small size, giving the engine its best chance to finish and catch:
| repo | fam | size | REGEX on vuln | ENGINE finding | cov | vuln region ranked | best taint | selected/deferred | wall |
|---|---|---|---|---|---|---|---|---|---|
@@ -229,70 +229,70 @@ Five public repos picked for flow-shaped families and small size (engine's best
| aws/bedrock-agentcore | injection | 3f | ✗ | **none** | incomplete | yes | — | 356 / 1009 | 305 s |
| miguelgrinberg/python-socketio | deser | 10f | **✓ (n=2)** | **none** | incomplete | yes | **tier1 score2.0 sink found** | 136 / **0** | 257 s |
**Result: regex 1/5, engine 0/5.** On a clean host, with budget to spare on most, the shipped engine **added no
Result: regex 1/5, engine 0/5. On a clean host, with budget to spare on most, the shipped engine **added no
catches over the regex floor.**
**Why (measured, traced — CORRECTED after inspecting `head.findings`, not ranker evidence):**
1. **The budget hypothesis is REFUTED.** banks, graphiti and socketio **finished their region queue** (deferred=0)
The diagnosis was corrected after inspecting `head.findings` rather than ranker evidence:
1. The budget hypothesis is refuted. banks, graphiti and socketio **finished their region queue** (deferred=0)
and still emitted no finding. Change-scoping to fit budget would not have changed the outcome.
2. **The engine produced ZERO vuln-family findings (path/injection/deser) on all 5.** Actual `head.findings`:
2. The engine produced no vuln-family findings (path/injection/deser) on any of the 5. Actual `head.findings`:
banks 1 (config_secrets), socketio 2 (config_secrets), launch-editor / graphiti / bedrock **0 findings each**.
The dataflow/taint tier traced **no source→sink flow** for the vuln families (`taint_evidence_not_applicable`;
e.g. banks found the `path.open` sink but **no source reaches it**). **Correction:** the tier=1 / score=2.0
"sink" seen in the ranker's `normalized_evidence` is the RANKER's static sink candidate, NOT a confirmed taint
finding — an earlier draft of this section misread it as detection partially working. It was not.
3. **`ambiguous_line_correspondence` is a downstream RED HERRING for these vulns.** It fires on all 5, but there
e.g. banks found the `path.open` sink but **no source reaches it**). Correction: the tier=1 / score=2.0
"sink" seen in the ranker's `normalized_evidence` is the ranker's static sink candidate, not a confirmed taint
finding. An earlier draft of this section misread it as detection partially working. It was not.
3. `ambiguous_line_correspondence` does not explain these missed vulnerabilities. It fires on all 5, but there
are **no vuln-family findings to anchor**, so a region-level witness fallback (the proposed "Option A") would
change nothing here. Witness/anchoring only touched the config_secrets findings that did appear.
4. **The LLM adjudication tier was DISABLED** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The
4. **The LLM adjudication tier was disabled** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The
rungs reached: `suspicion=0, model_corroborated=0, model_entailed=0–2, execution_confirmed=0`. The tier designed
to confirm a vuln the static dataflow cannot trace **never ran**.
5. **Ranking also under-promotes the vuln region** (tier 0 for most families; banks path family stuck low). So even
where a sink exists, the region is not analysed deeply enough to attempt a flow.
6. **Measurement caveat:** the pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe
6. The pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe
uses tight vuln-introducing diffs. Secondary to (2)/(4).
**Reprioritised engine levers (CORRECTED — anchoring is NOT the lever):**
- **#1 the LLM adjudication tier (disabled in the push path) — now the primary lever.** regex+dataflow = 1/5 with
the model OFF; the static tiers produce no vuln-family findings on these shapes. The product thesis
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot is the empirical case to
turn it on: **L1** model latency + calls/push, **L2** recall LIFT on the same 5 (and beyond). Costs money — stage
it (5–10 sizing → ~10–20 pilot → full only on lift).
Reprioritised engine levers (corrected, anchoring is not the lever):
- #1: the LLM adjudication tier (disabled in the push path) has the highest priority. regex+dataflow = 1/5 with
the model OFF; the static tiers produce no vuln-family findings on these shapes. The hypothesis
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot supports
testing it: **L1** model latency + calls/push, **L2** recall lift on the same 5 (and beyond). These calls cost money, so stage
them (5–10 sizing → ~10–20 pilot → full only on lift).
- **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the
vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0).
- **#3 change-anchoring robustness** — real, but only matters AFTER the static/LLM tiers produce vuln-family
- #3 change-anchoring robustness matters only after the static/LLM tiers produce vuln-family
findings to anchor. Deprioritised until (1)/(2) yield findings.
**Release read:** the pre-push net is NOT releasable on detection. Clean host, generous budget, LLM off: regex 1/5,
engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the decisive untested lever
is the **LLM adjudication tier**. Next: an **LLM-tier pilot (L1/L2)** on these 5 repos — does enabling the model
produce vuln-family findings regex+dataflow miss, and at what $/latency.
Detection does not meet the pre-push net's release requirements. Clean host, generous budget, LLM off: regex 1/5,
engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the untested lever
is the **LLM adjudication tier**. Next, run an **LLM-tier pilot (L1/L2)** on these 5 repos to test whether the model
produces vuln-family findings regex+dataflow miss, and measure its $/latency.
## ROOT CAUSE of engine 0/5 + the "#2 guided by #1" workstream (2026-10-12)
**Why the taint tier traced no flow (verified in code + evidence):** the pilot repos are **libraries**, whose
Code and evidence show why the taint tier traced no flow. The pilot repos are **libraries**, whose
untrusted input arrives as **public-API function parameters**, not HTTP requests. The Python taint **source**
vocabulary (`ruleset/semantic/python.toml`) is entirely web/CLI: Flask/Django `request.*`, `sys.argv`, `sys.stdin`,
`input(`. There is no source for a library's own API parameter. The engine *does* have a parameter-as-source mode
(`_parameters_are_input`, `scan.py`), but it fires **only for recognised entry points** (`_is_entry_point` and
`entry_kind != "function"`) — the pilot vuln regions carry `entry:false`, so their parameters are **deliberately
`entry_kind != "function"`). The pilot vuln regions carry `entry:false`, so their parameters are **deliberately
not** sources. That restriction is intentional: treating every function parameter as a source caused a flood of
false SQL-injections on WordPress/PMPro (documented in the `_parameters_are_input` docstring). So **the static
engine cannot safely widen its sources to library APIs without reintroducing that false-alarm explosion** —
`banks` path: sink `path.open` found, `source_local/near=false`; `graphiti`/`bedrock` injection: no source.
false SQL-injections on WordPress/PMPro (documented in the `_parameters_are_input` docstring). The static
engine cannot safely widen its sources to library APIs without reintroducing those false alarms.
For `banks` path: sink `path.open` found, `source_local/near=false`; `graphiti`/`bedrock` injection: no source.
**This is the precise seam for the maintainer's "#2 guided by #1" directive.** The static taint cannot *safely*
The maintainer's "#2 guided by #1" directive addresses this gap. Static taint cannot *safely*
decide which library parameters are trust boundaries; the **LLM can** reason about which public-API surface is
attacker-reachable (intent, not pattern). So:
- **#1 (LLM) guides:** on a changed library region the engine ranks but cannot source, the LLM adjudicates whether a
- #1 (LLM) guides: on a changed library region the engine ranks but cannot source, the LLM adjudicates whether a
parameter is an untrusted trust boundary and names the source→sink it believes exists (search-with-proof).
- **#2 (taint) traces + proves:** the engine then runs the flow from the LLM-named source to the known sink,
confirming (or refuting) with a traced path — a proof, not the LLM's say-so. Net: catch without the static FP
blowup, and every catch carries a dataflow witness.
- **Feedback:** where the LLM repeatedly names the same source shape (e.g. a registry-key → path family), promote it
- #2 (taint) traces + proves: the engine then runs the flow from the LLM-named source to the known sink,
confirming or refuting it with a traced path. The trace provides proof beyond the LLM's judgment. This would avoid the static FP
increase while giving every catch a dataflow witness.
- Where the LLM repeatedly names the same source shape (e.g. a registry-key → path family), promote it
to a static fact in `python.toml` guarded by the entry/shape conditions that keep WordPress quiet.
**Proposed first increment (pending budget confirmation — LLM calls cost money):** wire the disabled LLM tier on the
Proposed first increment (pending budget confirmation, LLM calls cost money): wire the disabled LLM tier on the
5 pilot regions in *LLM-guided-source* mode and measure: (a) does it name the right trust-boundary source per vuln,
(b) does the taint engine then trace source→sink and emit a vuln-family finding, (c) catch lift over regex (1/5) and
false-alarm cost on the ordinary changes. Stage it (5–10 sizing → ~10–20 → full on lift). This makes "#2 guided by
@@ -303,79 +303,78 @@ yields findings.
Sized the LLM tier on banks (the clearest pilot miss) with the project's own DeepSeek client (`deepseek-flash`).
**Infra + cost (sizing answer):** DeepSeek reachable, model id `deepseek-flash` valid. **~$0.00016–0.00025 per
call, ~1.1 s latency.** Sub-cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback
(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param — a client fix (don't send
DeepSeek is reachable, and model id `deepseek-flash` is valid. **~$0.00016–0.00025 per
call, ~1.1 s latency.** This costs less than a cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback
(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param. A client fix (don't send
`thinking` to that endpoint) is needed before it can be used; DeepSeek works today so it is not blocking.
**"2 guided by 1" validated as a mechanism — the LLM gives the correct verdict WHEN given the real sink + source
context:**
- Asked about `_load` (the pool's SZZ-labelled site, 135–141): `vulnerable=false` — *correct*, `_load` reconstructs
"2 guided by 1" works in this case: the LLM gives the correct verdict when given the real sink + source
context:
- For `_load` (the pool's SZZ-labelled site, 135–141), the model returned `vulnerable=false`. This is correct: `_load` reconstructs
paths from its own index, not the sink.
- Asked about the REAL sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the
context that `prompt.name` can be request-derived: **`vulnerable=true`** — *"prompt.name is joined into a
filesystem path and written without sanitization… path traversal or arbitrary file write."* A correct catch.
- Asked about the real sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the
context that `prompt.name` can be request-derived: **`vulnerable=true`**, *"prompt.name is joined into a
filesystem path and written without sanitization… path traversal or arbitrary file write."* This is a correct catch.
**Two blockers to realising it in the live pipeline (both now concrete):**
Two blockers to realising it in the live pipeline (both now concrete):
1. **Context, not single-shot.** The model only catches with the real sink + caller/source context. A single-shot
question on the ranker's region (which pointed at `_load`, and at the benign local-glob `_scan`) does not. This
requires the **agentic search-with-proof hunter** (explores callers/entry points to establish the trust
boundary), not the single CANDIDATE_QUESTION path. That is more calls → the real cost driver to size next.
2. **Corpus-label soundness.** The pool's banks label is **mislocated**: `head_lines=[[30],[104],[138–141]]`,
`function=_load` — the real sink is line **44**, outside head_lines±10 and a different function. So (a) even a
correct finding at line 44 would NOT score as a catch under `location_match`, and (b) the sink pre-exists in base
(the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet** — the pool even
boundary), not the single CANDIDATE_QUESTION path. That needs more calls → their cost is the next measurement.
2. **Corpus-label soundness.** The pool's banks label is mislocated: `head_lines=[[30],[104],[138–141]]`,
`function=_load`. The real sink is line **44**, outside head_lines±10 and a different function. So (a) even a
correct finding at line 44 would not score as a catch under `location_match`, and (b) the sink pre-exists in base
(the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet**. The pool even
annotates this bias ("delta hook can correctly stay quiet"). **Part of the pilot's engine 0/5 is this SZZ
mislocation, not a detection failure.**
**Revised next steps (budget approved, stay staged):**
The budget is approved. Keep the revised next steps staged:
- **Validate the 5 pilot labels first** (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair
targets. This separates true detection gaps from SZZ artifacts before spending on detection runs.
- **Then run the AGENTIC hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
- **Then run the agentic hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter
is the real "#1".
- Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2).
## Prompt evolution is a CORE lever for the LLM arm (maintainer directive 2026-10-12)
**"What is key is that we evolve the prompts as well"** — and **"GEPA or comparable; this is an evolving field."**
The bedrock hunter result makes this concrete: the agentic hunter ran (64 model calls, $0.0076) and reasoned
carefully, but with its **current prompts** it flagged adjacent concerns (network-PUBLIC default, download_file)
and never named the actual `install_packages` command injection — a *prompting/targeting* miss, not a cost or
capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence
outputs, evolves the prompt, and retries — built on **DSPy** (its GEPA / MIPRO optimisers), kept **pluggable**
behind an interface because the field moves fast (GEPA, MIPRO, TextGrad, successors). This aligns with
[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to BOTH LLM
uses here — the search-with-proof detector and the corpus-label auditor.
**"What is key is that we evolve the prompts as well"**, and **"GEPA or comparable; this is an evolving field."**
The bedrock hunter ran (64 model calls, $0.0076), but its **current prompts** led it to flag adjacent concerns
(network-PUBLIC default, download_file). It never named the `install_packages` command injection. This was a
*prompting/targeting* miss, rather than a cost or capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence
outputs, updates the prompt, and retries. Build it on **DSPy** (its GEPA / MIPRO optimisers). Keep it **pluggable**
behind an interface because the field changes quickly (GEPA, MIPRO, TextGrad, successors). This aligns with
[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to both LLM
uses here: the search-with-proof detector and the corpus-label auditor.
**The optimisation objective is to learn WHAT HINTS/ANNOTATIONS the model needs** to reach the real sink — not
just reword the prompt. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package
The optimisation must learn which hints/annotations the model needs to reach the real sink. Rewording the
prompt alone is insufficient. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package
names → command injection"; banks needed "`prompt.name` is request-derived and joined into a filesystem path →
traversal". DSPy optimises which evidence/hints to feed the signature so the model targets the true sink instead of
adjacent noise.
**Source those hints from the framework knowledge the project ALREADY ships** — the semantic facts taxonomy
(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework WITHOUT
Use the project's existing framework knowledge for those hints. The semantic facts taxonomy
(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework without
user annotations ([[php-detection-is-engine-only]] zero-annotation value). Where framework knowledge covers the
API, feed it as a hint; where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary
and, on repeated confirmation, it graduates into the facts (the #2 feedback loop). No new annotation burden on the
user. This is now a first-class part of the "#2 guided by #1" workstream, not a deferred nicety. (DeepSeek
API, feed it as a hint. Where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary,
which becomes a fact after repeated confirmation (the #2 feedback loop). No new annotation burden on the
user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek
`deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.)
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12) — 47% of labels are sound
Widened the banks label-mislocation check across all **49 public** pool vulns via DeepSeek (advisory + head code +
The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code +
pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
- **label_sound = 23/49 (47%)**; 53% unsound (25 pre-exist in base, 16 sink outside changed lines). Cost $0.038.
- By family (sound/total): injection 10/13, access_control 5/8, untrusted_destination 4/7, deserialization 1/2,
config_secrets 1/4, **path 1/7, output_encoding 1/8**. Injection labels mostly sound; path/output_encoding mostly
mislocate or pre-exist.
- **record 07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook** — on the ~53%
- Record 07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook. On the ~53%
pre-existing/mislocated vulns a change-scoped hook correctly stays quiet. The honest catchable denominator is the
sound subset; record 07 needs this caveat + a sound-subset recomputation.
- **Fair pilot re-score:** only bedrock + graphiti (injection) of the 5 are sound; the engine missed both — the
real detection gaps. The 23 sound targets are the fair local eval set for "#2 guided by #1".
- Only bedrock + graphiti (injection) of the 5 pilot cases have sound labels. The engine missed both, so these
are detection gaps. The 23 sound targets are the fair local eval set for "#2 guided by #1".
- Next: run the agentic search-with-proof hunter on the sound targets (start bedrock + graphiti), size its
per-vuln call count/cost, measure catch + false-alarm. Then recompute record 07 on the sound subset.
@@ -384,13 +383,13 @@ pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-1
Ran `ousast scan --mode deep` (DeepSeek `deepseek-flash`, model layer + tool_hunter, 60-call/8-hotspot bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed. Artifact:
`benchmarks/measurements/2026-10-12-hunter-sound-targets/`.
- **hunter 1/2 vs static engine 0/2.** graphiti: **HIT** — `falkordb_driver.py:207`, tag `family:cypher-injection`,
- **hunter 1/2 vs static engine 0/2.** graphiti: **HIT**, `falkordb_driver.py:207`, tag `family:cypher-injection`,
rationale traces `build_fulltext_query` concatenating user input into a Cypher string at line 278 (inside the
labelled function, location-matches). bedrock: **miss** — 12 suspicions on adjacent concerns, never named the
labelled function, location-matches). bedrock: **miss**, 12 suspicions on adjacent concerns, never named the
`install_packages`→`pip` command injection.
- **Cost is a non-issue:** ~$0.007/repo, 64 model calls for a whole-repo deep scan.
- **Everything stopped at `suspicion`** — the proof arm did not elevate to corroborated/entailed/execution_confirmed.
Release-grade BLOCK needs *proven* findings; measuring the proof arm's lift is the next depth.
- Cost is low: ~$0.007/repo, 64 model calls for a whole-repo deep scan.
- Everything stopped at `suspicion`. The proof arm did not elevate to corroborated/entailed/execution_confirmed.
Release-grade BLOCK needs *proven* findings; the next step is to measure the proof arm's lift.
- bedrock's miss is the concrete case for the DSPy prompt/hints loop above.
Next: (a) measure the proof arm (can it elevate graphiti's suspicion to proven?); (b) stand up the DSPy hint loop
@@ -399,37 +398,37 @@ run for the release-relevant shape.
## STRATEGIC PIVOT: core problem = unprovable reachability; FIX THE EVALUATION TARGET FIRST (2026-10-12)
**Core problem (named with the maintainer):** the tool can SPOT dangerous ops but cannot prove they are
ATTACKER-REACHABLE — the trust boundary lives outside the changed library code, in the caller/application a
change-scoped scan does not see; and the project rightly won't emit unproven alarms. Recall → ~0 from unprovable
reachability, not blindness. Cost/latency/engine-wall/corpus-labels were all downstream noise.
The maintainer and team identified the problem: the tool can find dangerous operations but cannot prove
an attacker can reach them. The trust boundary is outside the changed library code, in a caller/application
that a change-scoped scan does not see. The project will not emit unproven alarms. Recall → ~0 from unprovable
reachability, not blindness. Cost/latency/engine-wall/corpus-labels did not explain this problem.
**Chosen bet (of three — fix-target / advisory-with-conditions / build-prover): FIX THE EVALUATION TARGET FIRST.**
A pre-push hook runs inside a developer's APPLICATION (entry points present, request→sink in-repo), but the pool is
The chosen bet is to fix the evaluation target first. The three options were fix-target / advisory-with-conditions / build-prover.
A pre-push hook runs inside a developer's application (entry points present, request→sink in-repo), but the pool is
library-heavy, so we measured the hardest/wrong case.
**Shape classification (LLM, 49 public vulns, $0.033 — `benchmarks/measurements/2026-10-12-pool-p1-shape-classification/`):**
reachability_in_repo True only 12/49 (24%); 37/49 are pure libraries. A legitimate catch needs sound label AND
reachability in-repo → **5/49 (~10%)**: graphiti, desktopcommandermcp, malla, document-merge-service, mcp-remote.
**Shape predicted the hunter result** (graphiti app-shaped→caught; bedrock library→missed). So record-07's 3/92 is
scored against a ~90%-illegitimate denominator — the near-zero catch is largely a TARGET MISMATCH.
Shape classification (LLM, 49 public vulns, $0.033, `benchmarks/measurements/2026-10-12-pool-p1-shape-classification/`):
reachability_in_repo True only 12/49 (24%); 37/49 are pure libraries. A valid catch needs a sound label and
in-repo reachability → 5/49 (~10%): graphiti, desktopcommandermcp, malla, document-merge-service, mcp-remote.
Repository shape predicted the hunter result (graphiti app-shaped→caught; bedrock library→missed). So record-07's 3/92 is
scored against a ~90%-illegitimate denominator. The near-zero catch largely comes from a mismatch between the targets and the tool.
**Next:** deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch rate on the corrected set; then
Next: deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch rate on the corrected set; then
stand up an app-shaped eval set from known-vulnerable apps (VAmPI/NodeGoat/DjanGoat/vibe-py) and recompute the
floor. Deferred bets (advisory-with-conditions, reachability prover) revisited once the fair-target numbers land.
floor. Revisit the deferred bets (advisory-with-conditions, reachability prover) once the fair-target results are available.
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy) — target-fix validated (2026-10-12)
Deep scan on the 5 app-shaped+sound targets (`benchmarks/measurements/2026-10-12-hunter-fair-targets/`):
- **3/5 detected**: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
overlay — "outbound.request reached from parameter" on readFileFromUrl; the param-as-source mode fired because an
entry point is present). **2/5 missed**: malla (DOM/template XSS) and dms (Jinja SSTI) — both in families
families.toml DOCUMENTS as limited (template-mediated escaping).
- vs the library-heavy pilot's ~0 → **the target mismatch was real and material.** On an app-shaped repo the engine
- 3/5 detected: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
overlay, "outbound.request reached from parameter" on readFileFromUrl; the param-as-source mode fired because an
entry point is present). 2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families
documented as limited in families.toml (template-mediated escaping).
- The library-heavy pilot found ~0 → this confirms that the target mismatch affected detection. On an app-shaped repo the engine
itself detects (parameter is a recognized source once the entry point is in-repo).
- **But everything stops at `suspicion`/`static_corroboration`** — none proven. Cost ~$0.007/repo.
- Everything stops at `suspicion`/`static_corroboration`. None are proven. Cost ~$0.007/repo.
**Strategy verdict:** bet #1 (fix the eval target) is validated as NECESSARY and MATERIAL but NOT SUFFICIENT.
Bet #1 (fix the eval target) is necessary and improves detection, but is insufficient on its own.
Detection appears on fair targets; release-grade BLOCK still needs (a) proof-arm elevation (suspicion→proven) and
(b) family coverage (template XSS/SSTI) + the DSPy hint loop for misses. Next: recompute a floor on an app-shaped
sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure the proof arm's suspicion→proven lift.
@@ -438,53 +437,53 @@ sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure
Measured whether the proof arm can elevate the 3 fair-target suspicions (graphiti, mcp-remote, dcmcp) to
`exploit_demonstrated`.
- **Correction:** the search arm's `FAMILY_ORACLES` (search/demo.py) differs from families.toml's `verifier`.
All 3 hits HAVE a runtime oracle: injection→(sql,command), untrusted_destination→(ssrf). (deserialization/
access_control/config_secrets have NONE — empty tuple → `no_oracle`.) So dcmcp IS provable in principle
- The search arm's `FAMILY_ORACLES` (search/demo.py) differs from families.toml's `verifier`.
All 3 hits have a runtime oracle: injection→(sql,command), untrusted_destination→(ssrf). (deserialization/
access_control/config_secrets have none, empty tuple → `no_oracle`.) So dcmcp is provable in principle
(earlier "unprovable" was a families.toml misread).
- **But proving is a runtime differential:** `search/verify.verify()` requires (1) an AUTHORED declarative demo
- Runtime differential proof through `search/verify.verify()` requires (1) an authored declarative demo
(start the app on a port, run exploit steps, fill oracle placeholders: sql→database_path, command→marker_dir,
ssrf→callback_url), (2) BOTH head AND base runnable with OFFLINE deps (`pip --no-index`, `npm ci --offline`),
ssrf→callback_url), (2) both head and base runnable with offline deps (`pip --no-index`, `npm ci --offline`),
(3) a sandbox differential "owned effect in 3/3 affected runs and 0/3 safe runs". Local host has a sandbox
(`isolation_mode: userns`); the plane supplies a task-boundary dispatcher.
- **The scan/push path reaches only `suspicion`/`static_corroboration`** (verification.py levels) — there is NO code
- The scan/push path reaches only `suspicion`/`static_corroboration` (verification.py levels). There is no code
path from a scan suspicion to `exploit_demonstrated`, and no caller wires the search Coordinator from the push
flow. So **proof-arm lift at push time = 0/3**: not because the vulns are unreal, but because proving is a
per-target, offline-build, runtime, differential operation that cannot run inside a pre-push hook. (Prior
per-pair demo attempts were mostly inconclusive on this exact wiring — app startup, offline deps, oracle plumbing.)
flow. So proof-arm lift at push time = 0/3: not because the vulns are unreal, but because proving is a
runtime differential operation with an offline build for each target that cannot run inside a pre-push hook. (Prior
per-pair demo attempts were mostly inconclusive on this exact wiring, app startup, offline deps, oracle plumbing.)
**Strategic consequence (resolves the three-bet fork):** proof/BLOCK is an ASYNC capability (CI / plane / triage
with authored demos + a runnable sandbox), NOT a pre-push one. At push time the honest ceiling is corroborated
suspicion. So:
- **Bet #2 (advisory-with-conditions) is the honest PRE-PUSH product**: surface corroborated suspicions ("change
This resolves the three-bet choice. Proof/BLOCK runs asynchronously (CI / plane / triage
with authored demos + a runnable sandbox). At push time, the strongest available evidence is corroborated
suspicion:
- Bet #2 (advisory-with-conditions) is the honest PRE-PUSH product: surface corroborated suspicions ("change
introduces a flow that looks like X; reachable if untrusted Y reaches it"), let the dev / coding-agent (who has
the app context) judge — matching the "oversee coding agents" framing and the suspicion-rung reality.
- **Bet #3 (the proof arm) is the ASYNC BLOCK lane**: run `search/verify` on the plane/CI for the suspicions worth
proving, with authored (eventually DSPy-authored) demos. Worth an empirical run on one tractable target
(mcp-remote) to confirm the arm demonstrates on a real unseen vuln — a bounded, separate effort.
the app context) judge. This matches the "oversee coding agents" purpose and the evidence available at the suspicion rung.
- Bet #3 (the proof arm) is the ASYNC BLOCK lane: run `search/verify` on the plane/CI for the suspicions worth
proving, with authored (eventually DSPy-authored) demos. Run a bounded, separate test on one tractable target
(mcp-remote) to confirm that the arm demonstrates a real unseen vuln.
## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
the engine ZERO path findings in the pilot). Same target, same CPG — only the search changes.
no engine path findings in the pilot). The target and CPG are unchanged; only the search changes.
- Injected the two annotations a per-target LLM steer would supply: **parameter-as-source**
(`parameter_sources=True`, so `prompt.name` counts as untrusted) + the **missing file-write sink**
(`write_text`/`read_text` added to path sinks).
- **Baseline (shipped search): 0 resolved `prompt → write_text` flows** — its 5 flow rows are all census/
- The shipped search baseline resolved 0 `prompt → write_text` flows. Its 5 flow rows are all census/
evidence (`sink=None`), matching the pilot's zero path findings.
- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
`from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear ONLY with
- Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
`from_prompt_path` (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear only with
the annotations.
- **Thesis demonstrated on a concrete case:** banks was not a wrong target; the search was unannotated. The
fix needed exactly **two** inputs — the param-source and the file-write sink — which are precisely what the
- The banks result supports the hypothesis: the target was valid, but the search lacked annotations. The
fix needed exactly **two** inputs: the param-source and the file-write sink. These are what the
per-target LLM steer (DSPy-optimised, framework-seeded) has to learn to supply. This is the first measured
instance of the "#2 guided by #1" loop closing on an unseen target.
- Caveats: (1) one row shows `source=path` not `prompt` — the param-source fires broadly, a precision knob for
the steering loop; (2) this is a TRACE, not the runtime differential proof (proof stays the async BLOCK lane).
instance of a complete "#2 guided by #1" loop on an unseen target.
- Caveats: (1) one row shows `source=path` not `prompt`, the param-source fires broadly, the steering loop needs to control that breadth to improve
precision; (2) this is a trace, not the runtime differential proof (proof stays the async BLOCK lane).
**Instrument note / filed bug:** the first MVP used the single-query `cpg.run("taint", …)` path and returned
0/0 — an INSTRUMENT FAILURE, not a result (`cpg query taint failed` on every query). Root cause filed in
The first MVP used the single-query `cpg.run("taint", …)` path and returned
0/0 because of an instrument failure, rather than producing a valid result (`cpg query taint failed` on every query). Root cause filed in
`bugs/taint-sc-single-query-signature-skew.md`: `request_params()` emits 5 keys (`guards`, `fixedOrigin`,
`originAnchors`, `quotedSanitizers`, `fieldParameterSources`) that `taint.sc`'s `exec` signature does not
declare, so both single-query sub-routes (CLI `--param` and session named-arg) reject them. The batch path
.kiro/specs/valid-detection-benchmark/brief.md
+85
View File
@@ -0,0 +1,85 @@
# Valid detection benchmark
Created 2026-10-12. This brief **supersedes the frozen-SZZ-pool evaluation** (`unseen-repo-evaluation`), which is retired.
SZZ guessed its labels and placed the vulnerability in the wrong location about half the time (audit: path 1/7,
output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Catch /
false-alarm numbers from that pool are not valid benchmark results. This brief defines a benchmark with
sound ground truth and metrics that engineers and managers can understand.
## The two questions the benchmark answers
A pre-push security check must answer two questions:
1. **If I'm about to push a real vulnerability, does it stop me?**
2. **If my change is fine, does it leave me alone?**
The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on,
as is the LLM-guided arm. The repositories are ones the tool has **never seen**.
## What makes the data valid
### Positive cases — "should catch" (real, verified, reachable vulnerabilities)
- Take the label from the security fix, never from SZZ. Use the code *just before* the fix, at the file/function
the fix changed. The fix establishes the vulnerability location without a blame walk-back or mislocation.
- Keep only app-shaped cases where the vulnerability is reachable from an entry point *inside the repo*. This
can be a web route or a request handler; a CLI argument also qualifies. Drop library-internal vulnerabilities
that depend on a caller outside the repo. A pre-push tool cannot see that trust boundary, so scoring it is unfair.
- Verify each case through LLM adjudication and a human spot-check. Confirm (a) that the pre-fix code is
vulnerable and (b) that the fix addresses it. Also confirm (c) that it is reachable in-repo.
### Negative cases — "should not alarm" (ordinary safe changes)
- Use everyday commits from the **same** application repositories that are not security fixes. They are plentiful
and cheap to collect. Almost every real push is ordinary, so false alarms determine whether developers keep
using the tool. The negative set must be much larger than the positive set.
### Held out
- None of these repositories were used to build, train, tune, mine or harvest the tool.
A few dozen vulnerabilities with sound labels and in-repo reachability, each verified, are better than hundreds
of SZZ guesses. SZZ is wrong about half the time. Curating this set takes manual work, supported by the LLM-audit
tooling that found the SZZ pool's defects. Dozens of good cases are enough.
## Metrics someone understands
| Metric | Plain-English question | Example |
|---|---|---|
| **Catch rate** (recall) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" |
| **False-alarm rate** | On safe changes, how often does it cry wolf? | "Flagged 3 of 100 ordinary pushes" |
| **Precision** | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" |
| **Proof rate** | Of what it flags, how much can it *demonstrate*, not just suspect? | "12 of 18 catches came with a working demonstration" |
Report false alarms as **"per 100 pushes"**, which reflects a developer's experience. Vulnerabilities are rare
among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it.
Report the first two metrics together:
> "On unseen application repos: catches 90% of real vulnerabilities while false-alarming on 3% of ordinary
> pushes."
This wording is easier for a manager to understand than "3/92, Wilson95 1.1–9.2%".
## The gold-standard metric — and the project's edge
**Proved catches** do not depend on labels. The search-with-proof / differential-exploit arm demonstrates the
bad effect on the vulnerable code and its absence on the fixed code. The demonstration *is* the ground truth.
The headline to aim for is:
> "On unseen app repos: caught and *proved* N of M real vulnerabilities, with K false alarms per 100 ordinary
> pushes."
The architecture is built to produce this kind of benchmark.
## Why the old approach was not feasible (recorded so it is not re-tried)
- SZZ ground truth is machine-guessed and ~half wrong → every catch/FP number off it is noise.
- Library-heavy corpus → most cases are unreachable in-repo, so a pre-push tool is scored on things it
structurally cannot see.
- Scoring regex-only (engine and LLM off) measured a tier nobody would ship alone.
- Automated harvesting prioritized pool size over sound ground truth and in-repo reachability.
## Open questions for the spec phase
- How many verified positives are enough for a credible catch-rate interval (target set size)?
- The negative:positive ratio that reflects real push traffic (for an honest false-alarms-per-100-pushes).
- Curation workflow: how LLM adjudication + human spot-check are recorded and frozen per case.
- Language/framework coverage of the positive set (be explicit about what is and isn't represented).
.kiro/steering/roadmap.md
+123 -122
View File
@@ -7,23 +7,23 @@ This section supersedes the ordering below it. The older sections stay as histor
### Destination
On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every
family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work
is split by what each place is good at:
family, and BLOCK only for families whose precision has been proven on repositories the system never saw.
Each environment handles different parts of the work:
- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks.
- The kube-ax cluster (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes) runs the Joern engine as AX Tasks.
No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store,
and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
- The VM or CI runner handles candidate selection and the decision engine's model calls (DeepSeek). It also hosts
the memory store and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
- **kind** stays the local development profile, with the same code and manifests.
### The question everything answers
Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every
phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide
whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:
The A/B experiments test whether the safety net works on repositories it has never seen. Every phase below must
answer that question. Almost every push on an unseen repository is an ordinary change. Two numbers measure whether
the safety net is useful, based on the hook's advisory output as a user sees it:
1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything.
1. How often does it flag real vulnerability-introducing changes in unseen repositories (catch rate)?
2. How often does it flag anything on ordinary changes in the same unseen repositories (false-alarm rate)?
Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.
@@ -37,17 +37,19 @@ Within-pair AUC on known fix pairs answers neither. It stays a development signa
| Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` |
| Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` |
Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did
not move a generalisation number. From here, infrastructure and engine work is done only when it removes a
bottleneck for the measurements below.
Correction recorded 2026-10-04: most work in the previous two days went into infrastructure and engine internals
without improving a generalisation number. Further work on those areas must remove a bottleneck for the
measurements below.
### Where we are (2026-10-11)
The G0 measurement is **done**: the unseen pool is **frozen**, the pipeline is hardened and **bucket-first**,
and the slice-1 baseline (record 07) ran on the **docker lane** — the sanctioned interim while the kube `ax`
lane is restored. G0 is exited; the kube `ax` lane is now a **G1** concern (the engine arm at scale), not a
baseline blocker. (Correction 2026-10-12: an earlier revision of this section said the baseline was "the one
remaining step, to run on kube" — that was stale; record 07 was completed on docker right after, see below.)
**The frozen-SZZ-pool scoring approach is not feasible. Its catch/false-alarm numbers are retired as a
metric (2026-10-12 maintainer call).** The SZZ-derived labels place the vulnerability in the wrong location
~half the time (audit: path 1/7, output_encoding 1/8 sound). Only ~24% of entries are app-shaped
(reachability in-repo). Catch/false-alarm figures from this pool are invalid as a baseline.
Record 07's "3/92" and every pool catch/FP number are **not quoted and not a release gate**. Before any
detection number is valid, the pool must have **sound labels** and entries classified as app-shaped or library.
Until then, no pool-scored metric is cited anywhere.
| Area | State | Evidence |
| --- | --- | --- |
@@ -56,25 +58,21 @@ remaining step, to run on kube" — that was stale; record 07 was completed on d
| Harvest | Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop `--filter=blob:none`, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; **GitHub-rate-limit bound, not CPU**. | 95bed70, 857263d, 21aebe8, 7b711a6 |
| Persistence | **Bucket-first, not local disk.** `ousast-corpus` is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and **replay outputs go to RustFS too**, not `~/ousast-results` on local disk. | `storage-streaming.md`, `bank.py` |
| Replay image | Provenance pinned. CI `engine-image.yml` builds `ousast-replay-task` (harness with `/app/benchmarks/unseen/replay_entry.py`) + `ousast-engine-task` (Joern hook), publicly pullable by digest. The old `ousast-runner:dev` (248 MB ax-runner, no `/app/benchmarks`, no Joern) is the **wrong** image for replay. | CI run 38116281676 |
| Baseline (G0.2) | **DONE — record 07 (`e901640`), docker lane.** 1300/1300 changes; catch 3/92 = **3.3%** (Wilson95 1.1–9.2%), false-alarm **3.1%** (cluster95 1.6–5.4%), instrument_failure 54 (4.2% < 5% guard). `engine_covered=0` and no model calls → this is a **regex/quick-rule floor** (the engine never ran under the ~200s pre-push wall; the LLM tier was disabled). Baseline row in `usage-p1.jsonl` (freeze_digest `3371c67d`) = **G0 exit**. Ran on docker (not kube `ax`) per the sanctioned interim. | `benchmarks/measurements/2026-10-11-unseen-07-slice1-baseline/` |
| Baseline (G0.2) | **RETIRED as a metric.** A replay (record 07) ran end-to-end on the docker lane, which proves the pipeline works — but its catch/false-alarm figures are **not a usable baseline** (scored against ~half-mislocated SZZ labels on a mostly non-app-shaped corpus) and are **not quoted or used as a gate**. The pool must be cleaned to sound labels first. | pipeline proven; metric not used |
Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives
in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next
pool or arm is cheap, not a rediscovery.
Sourcing and engine-image work took time. The pipeline and these docs now record bucket-first persistence and
pinned image provenance, along with the measured yield. The next pool or arm can reuse that work.
### Readiness plan (G0 exited; next is G1)
### Readiness plan — build a VALID benchmark first
**G0 exit = record 07 is DONE** (docker lane, regex/quick-rule floor; see the Baseline row above). Steps 1–4
below are complete. The kube `ax` lane was NOT needed for the baseline and is now a **G1** target.
The frozen-SZZ-pool approach is retired (above). The replay pipeline works end-to-end, and the harvest/freeze/bucket
machinery remains available. The engine/replay images are pinned. No catch/false-alarm figure from that pool is used.
1. ~~Kube replay lane~~ — **not needed for G0**; the baseline ran on the docker lane (the sanctioned interim).
The `ax` lane moves to G1 (see below).
2. ~~Unblock the replay~~ — done; replay ran over the frozen slice.
3. ~~Run slice-1 baseline~~ — done: `--lane docker --arm default`, 1,300 changes, no model cost, outputs to RustFS.
4. ~~Adjudicate (7.3) → record 07 (7.4)~~ — done = **G0 exit** (`usage-p1.jsonl` freeze_digest `3371c67d`).
5. **Next — G1 arms on the frozen pool.** Re-score the arms that the regex floor cannot be (engine evidence,
DeepSeek decision engine, combiner) to see what beats catch 3.3% at a fixed false-alarm budget. This is where
the kube `ax` lane is needed, because the engine arm cannot run under the ~200s pre-push wall on a single host.
Next, define and build the valid benchmark. Use sound, verified positives with in-repo reachability and ordinary
negatives on unseen repos. Measure the whole tool with metrics an engineer understands. The design is in
`.kiro/specs/valid-detection-benchmark/brief.md`. Only once that benchmark exists does any detection number
(catch rate, false-alarms-per-100-pushes, proved-catch) mean anything. The kube `ax` lane is still the place the
engine tier runs at scale, but it is infrastructure, not the gate.
### Restoring the kube `ax` lane (for G1, operator-owned)
@@ -82,10 +80,10 @@ The AX executor is **built and smoke-tested** (`benchmarks/ax/batch.py`, `engine
lanes, OOM→docker fallback; an engine smoke test dispatched to the 2-worker cluster end-to-end on 2026-10-04).
What remains before the G1 engine-arm replay:
- **Use the working `ax` binary: `~/go/bin/ax`** — NOT `~/.local/ax-v0.3.1/ax`, which hits the AX-lane blocker
- Use the working `ax` binary, `~/go/bin/ax`. `~/.local/ax-v0.3.1/ax` hits the AX-lane blocker
(recorded in the unseen spec `tasks.md`, 2026-10-11). The coordinator commands in the READMEs are being
corrected to this binary.
- **Store endpoint:** `benchmarks/ax/batch.py` expects a public-HTTPS DNS, presign-capable S3 endpoint reachable
- `benchmarks/ax/batch.py` expects a public-HTTPS DNS, presign-capable S3 store endpoint reachable
from the cluster; the cluster carries no model/store credentials (it reads/writes via short-lived presigned
links). This and the cluster's store access are operator/AX-owned.
- **Run the open spec tasks** that gate G1: **4.4** (engine smoke on kube-ax → record 04) and **5.4** (replay
@@ -94,23 +92,27 @@ What remains before the G1 engine-arm replay:
### Phases
**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`)
**G0. A VALID detection benchmark** (spec `valid-detection-benchmark`; supersedes the retired
`unseen-repo-evaluation` frozen-SZZ-pool approach)
1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population.
Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where
possible) and ordinary commits from the same history.
2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and
record catch rate and false-alarm rate, with confidence intervals, per family and overall.
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.
1. Use real vulnerabilities labelled *from the fix*: the pre-fix code at the file/function the fix changed,
without an SZZ blame-walk. Keep only those reachable from an in-repo entry point. Confirm each case through
LLM adjudication and a human spot-check so the positives are sound, app-shaped and verified.
2. Ordinary negatives: everyday non-fix commits from the same application repos, a set much larger than the
positives (false alarms discourage adoption).
3. Measure the **whole tool** (all tiers on) on **unseen** repos, and report metrics an engineer reads: catch
rate, **false-alarms per 100 pushes**, precision, and **proved-catch** (demonstrated, label-free). See the
brief for the full definition and why the SZZ pool was not feasible.
Exit: the pool is frozen and the current production hook has a measured baseline on it.
Exit: the benchmark exists (frozen, verified) and the whole tool has a first measured result on it.
**G1. Re-score the arms we have on the pool**
**G1. Score the arms on the valid benchmark**
Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is
an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.
Quick rules, engine evidence, the LLM-guided arm, and their combination, each a registered decision rule,
scored by catch rate at a fixed false-alarms-per-100-pushes budget, with proved-catch where available.
Exit: one measured table of arms on unseen code; keep only what generalises.
Exit: one measured table of arms on unseen app-shaped code; keep only what generalises. (No number is quoted
until it comes from this benchmark, never the SZZ pool.)
**G2. Improve what moves the unseen-repo numbers**
@@ -120,63 +122,63 @@ changes, more labelled pairs from more distinct repositories.
Active G2 workstreams (2026-10-11; progress tracked in `.kiro/specs/engine-change-scoped-analysis/brief.md`):
**#1 priority after record 07: get real (semantic) detection under the pre-push budget.** The record-07 baseline
is **regex only** — verified: the push arms hardcode `model="disabled"`, `max_model_calls=0`, and the engine never
ran (`engine_covered=0`); measured `catch≈2.5%`. Pattern rules alone will never be enough (else `grep` would do).
The two tiers that carry detection both sat out: the **engine** (Joern dataflow) and the **LLM adjudication**
(disabled in the push path). The workstream is an **efficiency barrier analysis** — a per-tier budget ledger
Priority #1 after record 07: run semantic detection within the pre-push budget. The record-07 baseline
is **regex only**. The push arms hardcode `model="disabled"`, `max_model_calls=0`, and the engine never
ran (`engine_covered=0`); measured `catch≈2.5%`. Pattern rules alone are insufficient; otherwise `grep` would do.
The two detection tiers did not run: the **engine** (Joern dataflow) and the **LLM adjudication**
(disabled in the push path). The workstream measures **efficiency barriers** with a per-tier budget ledger
against a realistic pre-push window (≤120 s target, 300 s hard):
- **Engine: E1/E2 DONE (2026-10-12, clean host) — see brief "E1/E2 RESULTS".** CORRECTION: the engine's
- Engine: E1/E2 DONE (2026-10-12, clean host), see brief "E1/E2 RESULTS". CORRECTION: the engine's
`query=0` wall was **host contention** (the baseline replay), not an intrinsic floor. On a clean host the
shipped engine **runs dataflow** (query 0→35 s, selected 0→**356 regions**). The evidence/taint batch (~225 s)
is **invariant to session and preload**; graph-load is cheap (~35 s incl `importCpg`). The real barrier is
`taint.sc`'s ~160 s per-batch flow work, so `build(~30)+evidence(~225)≈255 s` of 300 s defers 1009/1365
regions. Lever = **change-scope the family/file set** the taint batch walks (NOT session/preload). E3 analysis
regions. Reduce the work by **scoping the family/file set to the change** (not session/preload). E3 analysis
per-pair cost; E4 build vs repo size remain.
- **ENGINE-ON PILOT DONE (2026-10-12, clean host, 5 public repos) — see brief "ENGINE-ON PILOT RESULTS".**
**regex 1/5, engine 0/5.** The engine added NO catches over the regex floor. The **budget hypothesis is
REFUTED**: 3/5 repos finished their queue (deferred=0) and still emitted no finding — so change-scoping is NOT
the path to catch. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser,
launch-editor, score 2.0), but **withholds any finding it cannot anchor 1:1 to changed lines**: all 5 hit
- **ENGINE-ON PILOT DONE (2026-10-12, clean host, 5 public repos), see brief "ENGINE-ON PILOT RESULTS".**
**regex 1/5, engine 0/5.** The engine added no catches over the regex floor. The budget hypothesis is
refuted: 3/5 repos finished their queue (deferred=0) and still emitted no finding, so change-scoping will not
improve catch rate. The engine ranked the vuln region in all 5 and found the right sink on 2 (socketio deser,
launch-editor, score 2.0), but withholds any finding it cannot anchor 1:1 to changed lines: all 5 hit
`ambiguous_line_correspondence` → `witness_unresolved` → `coverage_status=incomplete`, suppressing even the
scored sinks. Confirmed NOT a shallow-clone artifact. **Release read: the net is NOT releasable on detection.**
- **CORRECTED after inspecting `head.findings`:** the engine produced **ZERO vuln-family findings** (path/
injection/deser) on all 5 — only config_secrets. The tier1/score2.0 "sink" was the RANKER's static candidate,
not a taint finding. `ambiguous_line_correspondence` is a downstream red herring (nothing to anchor), so the
scored sinks. This is not a shallow-clone artifact. Detection does not meet release requirements.
- Corrected after inspecting `head.findings`: the engine produced no vuln-family findings (path/
injection/deser) on all 5, only config_secrets. The tier1/score2.0 "sink" was the ranker's static candidate,
not a taint finding. `ambiguous_line_correspondence` does not explain the missing findings (nothing to anchor), so the
region-level witness fix (old "Option A") was **refuted before implementation**. The **LLM tier was disabled**
(model_calls=0). **ROOT CAUSE:** pilot repos are LIBRARIES whose untrusted input is public-API parameters;
(model_calls=0). The cause is that pilot repos are libraries whose untrusted input is public-API parameters;
Python taint sources are web/CLI only, and the param-as-source mode fires only for entry points (vuln regions
are `entry:false`) — widening it statically would reintroduce the WordPress/PMPro false-SQL-injection flood.
**Workstream = maintainer's "#2 guided by #1":** the LLM adjudicates which library parameter is a trust boundary
are `entry:false`). Widening it statically would reintroduce the WordPress/PMPro false-SQL-injection flood.
The workstream follows the maintainer's "#2 guided by #1" directive: the LLM adjudicates which library parameter is a trust boundary
(static rules can't without FPs), the taint engine then traces that source→sink as proof, and repeated source
shapes get promoted to guarded static facts. First increment (PENDING BUDGET): wire the disabled LLM tier on the
shapes get promoted to guarded static facts. First increment (pending budget): wire the disabled LLM tier on the
5 pilot regions in LLM-guided-source mode; measure source naming, traced finding, catch lift vs regex 1/5, and
false-alarm on ordinary changes; stage 5–10 → ~10–20 → full on lift. #3 change-anchoring deprioritised until
tiers yield findings.
- **BANKS FIXED-SEARCH MVP DONE (2026-10-12) — "#2 guided by #1" closes on an unseen target.** See brief
- **BANKS FIXED-SEARCH MVP DONE (2026-10-12), "#2 guided by #1" closes on an unseen target.** See brief
"BANKS FIXED-SEARCH MVP" + `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. On banks (the
library that gave the engine zero), injecting the two annotations an LLM steer would supply (param-as-source
+ `write_text`/`read_text` sink) makes the engine resolve `prompt → path-join → write_text` where the
shipped search resolved nothing. Proves the governing principle "no wrong targets, only wrong searching" and
names the exact inputs the DSPy-optimised steer must learn. Filed instrument bug:
shipped search resolved nothing. This supports the principle "no wrong targets, only wrong searching" and
identifies the inputs the DSPy-optimised steer must learn. Filed instrument bug:
`.kiro/specs/engine-change-scoped-analysis/bugs/taint-sc-single-query-signature-skew.md` (single-query taint
path dead on an exec-signature skew; batch path unaffected).
- LLM: L1 model latency and calls per push; L2 the recall LIFT of the LLM tier over regex+dataflow on the pool.
- LLM: L1 model latency and calls per push; L2 the recall lift of the LLM tier over regex+dataflow on the pool.
- Deliverable: a barrier table (budget × tier combo → fits? measured recall/FP) → the minimal releasable config.
- **Change-scoped engine analysis — DEMOTED again by the pilot.** Branch `engine-change-scoped-evidence-pass`
- The pilot lowers the priority of change-scoped engine analysis again. Branch `engine-change-scoped-evidence-pass`
(unmerged). Fitting the budget does not produce catches (3/5 pilot repos finished the queue with zero findings),
so this is not on the catch critical path. Keep only as a latency optimisation once detection works.
so this is not required to improve catch rate. Keep only as a latency optimisation once detection works.
- **Line-correspondence (merged c39fbb8).** `ambiguous_line_correspondence` fires on 5/5 realistic diffs, but the
pilot's corrected diagnosis shows it is NOT the detection blocker here — the static tiers produce no vuln-family
findings to anchor in the first place. A region-level witness fallback only matters once #1/#2 yield findings.
pilot's corrected diagnosis shows it does not block detection here. The static tiers produce no vuln-family
findings to anchor. A region-level witness fallback only matters once #1/#2 yield findings.
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.
**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
G5. Qualification and rollout: adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
BLOCK only for families that passed.
### Cross-cutting rules
@@ -214,13 +216,13 @@ progress, not adopted). See `overview.md` for the current-state summary.
Build an actionable pre-push security safety net for AI-accelerated development.
The maintainer accepted the following order: **prove one useful detection, make it
fast, prove cross-language transfer, qualify independently, then enable narrow advisory
fast, prove cross-language transfer, qualify independently, then enable a narrow advisory
rollout**. The executable outcome contract lives in the existing
[pre-push-safety-net release milestone spec](../specs/pre-push-safety-net/release-milestones.md).
**We are at M1b. M1a is met; no other release milestone is verified yet.** The published
`v1.2.0-alpha.1` and 28/29 completed implementation tasks describe delivered machinery,
not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
`v1.2.0-alpha.1` and 28/29 completed implementation tasks show that the machinery is implemented.
They do not demonstrate a useful safety net. Task 8.3 and rollout remain NO-GO.
| Order | Milestone | Observable exit | Current state |
|---|---|---|---|
@@ -252,26 +254,26 @@ not dropped safety checks. Advisory and blocking share the same actionability ba
Graph persistence now works: identical-tip replays take 18.5–18.9s. All 18 other measured
transactions still time out and 0/24 target checks complete. A local Joern session census
probe is promising, but full-query equivalence and hook performance are unverified.
probe shows promise, but full-query equivalence and hook performance are unverified.
These findings inform M2; they do not complete M1. No second IR/engine, new heuristic
scope selector, differential graph mutation or unrelated backlog is part of this path.
## Existing Spec Updates
- [ ] pre-push-safety-net — Execute the accepted M1–M5 release milestone amendment.
- [ ] pre-push-safety-net, Execute the accepted M1–M5 release milestone amendment.
Owns immutable replay, comparison, admission, evaluation and reporting. Preserve the
existing completed tasks; task 8.3 remains blocked until qualification passes.
- [ ] contributor-scan — Supply the required query/operation/context fixes for M1,
- [ ] contributor-scan, Supply the required query/operation/context fixes for M1,
followed by equivalent bounded session execution for M2 and shared transfer evidence.
Frontend retention 2.18 and named census repair 2.19 are already verified.
- [ ] flow-aware-ranking — Preserve ranker-owned scope and normalized cross-language
- [ ] flow-aware-ranking, Preserve ranker-owned scope and normalized cross-language
behavior. New ranking research is not a prerequisite for the first useful comparison.
- [ ] zero-with-a-reason — Adjacent diagnostics work; finishing its whole backlog is
- [ ] zero-with-a-reason, Adjacent diagnostics work; finishing its whole backlog is
not required before M1.
## Specs (dependency order)
- [ ] pre-push-safety-net — Existing approved implementation, dependent on contributor-scan.
- [ ] pre-push-safety-net, Existing approved implementation, dependent on contributor-scan.
Groups 1–7 and tasks 8.1, 8.2, 8.4 are verified; 26/27 executable tasks complete.
Remaining release work follows [M1–M5](../specs/pre-push-safety-net/release-milestones.md).
This amendment creates no duplicate implementation spec and resets no prior approvals.
@@ -289,17 +291,16 @@ supersede this accepted sequence. Blocking deployment follows a later explicit d
The sections below preserve prior plans and measurements. Statements about the current bottleneck,
PHP coverage, rollout order or feature readiness are superseded by the current section above.
**Rewritten 2026-09-09.** The previous version (kept as `.kiro/archive/steering/roadmap-2026-09-04.md`, outside steering) described a pre-Joern world —
**Rewritten 2026-09-09.** The previous version (kept as `.kiro/archive/steering/roadmap-2026-09-04.md`, outside steering) described a pre-Joern world,
"adjudication is still Python-`ast` plus a tree-sitter CLI stub", "Joern-as-required deferred". Joern was
adopted, the model layer was built, wired and measured, and two repositories have been scanned. See
adopted. The model layer was built and wired, then measured. Two repositories have been scanned. See
`overview.md` for what is true today; this is what happens next and in what order.
## The target: a first release for small projects
> "a first release that is ready for smaller projects like libpng or wordpress"
Those two are, today, exactly the cases that do not work — and they fail for **different** reasons, which is
what makes them a good pair of targets rather than one.
Neither works today. They fail for different reasons, so both are useful targets.
| target | why it fails now | what a release needs |
| --- | --- | --- |
@@ -312,8 +313,8 @@ So "ready for" has to mean two different things, and the release criteria say wh
* For one it **cannot** (libpng today): run inside its envelope, report nothing false, and *state the limit*
rather than returning a quiet clean bill of health.
The second is not a consolation. A tool that says "I do not model this class" is usable; one that silently
finds nothing is the thing that teaches developers to distrust SAST.
A tool that says "I do not model this class" is usable. Silent failure to find anything teaches developers to
distrust SAST.
### Release criteria (v0.1)
@@ -321,15 +322,15 @@ finds nothing is the thing that teaches developers to distrust SAST.
2. **Zero known-false entailed findings** on every pinned checkout.
3. Finds the known vulnerability wherever a family can decide it in principle (Req 4.6).
4. States what each family cannot decide, in the report (Req 5.6).
5. The report is proportionate — no rule repeating itself past a stated count (Req 5.5).
5. The report is proportionate: no rule repeats itself past a stated count (Req 5.5).
6. Installs and runs without the user having Joern, a JVM, or a PHP interpreter (docker compose + wrapper).
7. A committed regression baseline per target, so the next change is measurable rather than argued about.
### What that reorders
PHP moves **up**: it is the shortest path from here to a target the maintainer named, and it is the case the
PHP moves up: it is the shortest path from here to a target the maintainer named, and it is the case the
existing arbiter is best suited to. The C decision moves up with it, because criterion 4 needs `c/memory`
either earning its place or being scoped — and scoping it is cheap.
either earning its place or being scoped, and scoping it is cheap.
## The one test everything is ordered by
@@ -337,40 +338,40 @@ either earning its place or being scoped — and scoping it is cheap.
> capabilities but never use it"
So the ordering principle is: **make the output worth reading, then make it find more.** A better arbiter
behind an unreadable report changes nothing, and a second language before the report is fixed just doubles the
behind an unreadable report changes nothing, and a second language before the report is fixed doubles the
noise.
## Now — make a repository scan finish at all
The output work this section used to hold is done, and PHP went considerably further than it planned: three
CVEs on real WordPress plugins, entailed by the graph with no model in the loop, each dropping on its fixed
side. The detector is not the problem any more. **Finishing is.**
The output work previously listed here is done. PHP detection exceeded the plan: the graph entailed three
CVEs on real WordPress plugins without a model, and each finding disappeared on the fixed side. The remaining
problem is getting scans to finish.
A 637-file plugin builds in 76 seconds and then the taint query does not complete. Everything below is
ordered by one question: what is the shortest path to a scan of a real plugin that reports its CVE?
1. **5.14 — stop asking questions that cannot have an answer.** 53% of taint requests are put to a family
1. For **5.14**, stop asking questions that cannot have an answer. 53% of taint requests are put to a family
whose sinks do not appear in scope; every one of PMPro's 487 `deserialization` requests is asked of a
repository that never calls `unserialize`. Arithmetic, not judgement, and it cannot lose a finding.
repository that never calls `unserialize`. This is an arithmetic check and cannot lose a finding.
4.6 hours → 2.2. Do this first because it makes every later measurement cheaper to run.
2. **5.12 — rank, because 84% of regions currently share one rank ordered alphabetically.** Two thirds of the
budget is spent arbitrarily. Fifty well-chosen regions beat five hundred; combined with (1) that is the
2. For **5.12**, add ranking. Currently 84% of regions share one rank and are ordered alphabetically. Two thirds of the
budget is spent arbitrarily. Fifty well-chosen regions are better than five hundred; combined with (1) that is the
difference between hours and minutes. The ranker is an LLM job, and it is safe as one because ranking
cannot manufacture a finding -- it decides what is looked at, never what is reported.
**The loop is closed and that is a design requirement, not a caveat:** a region the ranker excludes is
cannot manufacture a finding. It decides what is looked at, never what is reported.
The loop is closed and that is a design requirement, not a caveat: a region the ranker excludes is
never arbitrated, so it never produces a label, so the ranker never learns it was wrong. An exploration
slice and a small fully-labelled corpus are what make the signal unbiased.
3. **5.13 — the 6.63 s/request itself**, if (1) and (2) are not enough. `callDepth=3` over 7,061 methods is
the term that makes a request cost seconds; the two-stage joins are NOT the cost (measured: 22%, and they
3. For **5.13**, reduce the 6.63 s/request itself if (1) and (2) are not enough. `callDepth=3` over 7,061 methods is
the term that makes a request cost seconds; the two-stage joins are not the cost (measured: 22%, and they
found two of the three CVEs).
Only after a scan finishes is there any point tuning what it finds.
## Next — the families that cannot be asked
4. **5.9 — PHP obligation facts.** `dominance_specs(language="php")` is empty, so missing-capability checks
4. For **5.9**, add PHP obligation facts. `dominance_specs(language="php")` is empty, so missing-capability checks
are not asked on any WordPress scan. Not a miss; nothing was measured.
5. **5.6 — a sanitizer cleanses the family it cleanses.** The list is flat, so `esc_sql` reads as sufficient
5. For **5.6**, associate each sanitizer with the family it cleanses. The list is flat, so `esc_sql` reads as sufficient
for XSS and `esc_html` as insufficient for SQL. Wrong in both directions, so no `output_encoding` verdict
currently means anything.
6. **File the php2cpg defect upstream.** A `global` inside a closure yields a CPG Joern's own dataflow
@@ -379,12 +380,12 @@ Only after a scan finishes is there any point tuning what it finds.
## Then — close the deferred engine questions
8. **2.14 — control dependence.** Absent from both CPGs, so every guard question runs on CFG dominance. Find
8. For **2.14**, check control dependence. It is absent from both CPGs, so every guard question runs on CFG dominance. Find
out whether Joern's CDG pass can be applied from the query script and what it costs. If it cannot, that is
a real bound on what this engine can reason about and should be recorded as one.
9. **2.13 — a destination allocated to size.** Deliberately blocked on 6, so the rule can be measured for what
9. **2.13** covers a destination allocated to size. It is deliberately blocked on 6, so the rule can be measured for what
it *silences* as well as what it clears.
10. **Group 3–4 — ship it.** Contributor output, the Docker image, the memory-family decision, retiring
10. Ship **Group 3–4**: contributor output, the Docker image, the memory-family decision, and retirement of
`evidence_level` now that the rung carries the meaning.
## Alongside — keep the codebase from growing into the problem
@@ -393,16 +394,16 @@ Only after a scan finishes is there any point tuning what it finds.
`.kiro/specs/ir-consolidation/` (brief, unapproved) separates two questions with opposite answers:
* **The source IR is already carried, not about to be added.** `semantic/` is 3,218 lines — 16% of the
codebase — holding a hand-written IR and flat taint engine measured at **2.2% entailment against the CPG's
* The source IR already exists in the project. `semantic/` is 3,218 lines, or 16% of the
codebase. It holds a hand-written IR and flat taint engine measured at **2.2% entailment against the CPG's
41.3%**, still running as a proposer. Vine and Ghidra P-Code are *binary* IRs and do not help here; the
work is subtraction. A candidate ~1,000-line deletion with a number attached.
work is subtraction. This is a candidate ~1,000-line deletion supported by a measurement.
* **The missing abstraction is C arithmetic**, and there an existing IR is exactly right. GhidraPAL's
three-valued-logic abstract interpreter over P-Code decides "can this index exceed this allocation", which
is the class `c/memory` cannot see and every libpng overflow is. Vine is dormant since 2009 and is not the
candidate.
The ordering constraint is the point: **question 1 lands before question 2 starts**, or adopting an IR makes
The order matters: **question 1 lands before question 2 starts**, or adopting an IR makes
the growth problem worse rather than better.
## After — the feedback loop
@@ -413,15 +414,15 @@ the growth problem worse rather than better.
## Not now, and why
- **`authorization-obligations`** (24 open tasks, approved) — the dominance arbiter delivered its substance by
- `authorization-obligations` (24 open tasks, approved): the dominance arbiter delivered its substance by
another route. It should be reconciled or retired rather than worked, and carrying it as "approved, not
started" misrepresents both.
- **`reachability-flow-model`** (23 open, nothing approved) — largely delivered by adopting Joern. Same
- `reachability-flow-model` (23 open, nothing approved) was largely delivered by adopting Joern. Same
treatment.
- **`constrained-detector`** (4 of 20, tasks unapproved) — parked.
- **More corpus volume.** 198 of 238 OWASP pairs were left unvendored deliberately: volume without
- `constrained-detector` (4 of 20, tasks unapproved) is parked.
- More corpus volume is deferred. 198 of 238 OWASP pairs were left unvendored deliberately: volume without
readability makes every number worse, and the corpus is not the binding constraint. Reachability is.
- **A second CPG engine, a server mode, an execution rung.** Server mode was measured and declined (6% of a
- A second CPG engine and an execution rung are deferred, as is server mode. Server mode was measured and rejected (6% of a
scan). The other two have no evidence calling for them yet.
## How to know it is working
@@ -430,6 +431,6 @@ The scoreboard in `overview.md`, and one question: **can a developer run this on
write and get something they act on?** Today that is true for one Python application and false for one C
library, and the specs above are ordered by what changes that.
A release is ready when that question is answered **yes for WordPress and honestly for libpng** — found for
the one whose class we decide, and a stated limit for the one we do not. Both are release criteria; only one
A release is ready when it answers **yes for WordPress and honestly for libpng**. It must find the vulnerability
where the engine can decide the class, and state the limit where it cannot. Both are release criteria; only one
of them is a detection claim.