mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 04:59:28 +00:00
docs: retire the SZZ-pool metric, add the valid-benchmark brief, plain-English pass
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in the wrong place about half the time, and ~76% of entries are library-internal (not reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise. Retired as a metric and a gate across the roadmap and briefs; no pool-scored number is quoted anywhere. Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified; ordinary negatives from the same repos; unseen repos; the whole tool measured; plain metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch). Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two active briefs via Codex, then reviewed the diff: only these files changed, every number/path/hash/command is unchanged, and the retired-metric decision is intact. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
c3f610f9f4
commit
b08f372022
@@ -1,93 +1,93 @@
|
||||
# Engine change-scoped analysis (G2 workstream)
|
||||
|
||||
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks a concrete lever for the
|
||||
engine-detection gap found while driving the G0 baseline. This is G2 ("improve what moves the unseen-repo
|
||||
numbers"), gated on the pool metric; it is NOT part of the G0 record-07 baseline (that runs on the quick-rule tier).
|
||||
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks work to address the
|
||||
engine-detection gap found during the G0 baseline run. This is G2 ("improve what moves the unseen-repo
|
||||
numbers"), gated on the pool metric; it is not part of the G0 record-07 baseline (that runs on the quick-rule tier).
|
||||
|
||||
## Problem (measured, not assumed)
|
||||
|
||||
On real unseen vulnerability changes the engine produces **no finding** — but the diagnosis (live, 300 s long-deadline
|
||||
replays) shows it is usually **not a detection-capability gap**:
|
||||
The engine produces **no finding** on real unseen vulnerability changes. Live 300 s long-deadline replays show
|
||||
that the cause is usually not a detection-capability gap:
|
||||
|
||||
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
|
||||
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
|
||||
`change_context_incomplete`. **Zero analyses ran.** And the ranked candidates were **repo-wide and unrelated to
|
||||
`change_context_incomplete`. No analyses ran. The ranked candidates were **repo-wide and unrelated to
|
||||
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
|
||||
re-audit the whole repo and runs out of budget before analyzing the change.
|
||||
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
|
||||
`contributor-scan ... projection_unavailable`. Multi-causal; correspondence was one part (fixed separately,
|
||||
commit c39fbb8), capability registry is Task 8.3.
|
||||
`contributor-scan ... projection_unavailable`. This has several causes. Correspondence was one (fixed separately,
|
||||
commit c39fbb8); the capability registry is Task 8.3.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
A pre-push hook should vet **the code this push changed**, not re-scan the whole repository. Seeding/scoping the
|
||||
engine's analysis to the **changed declarations and their dataflow neighborhood** (the `change_sites` are already in
|
||||
the replay spec) should let analysis **actually run within budget** instead of being deferred — which is where
|
||||
detection is being lost, not in Joern's capability.
|
||||
A pre-push hook should check **the code this push changed**. Starting the engine's analysis from the
|
||||
**changed declarations and their dataflow neighborhood** should let it run within budget instead of being
|
||||
deferred. The `change_sites` are already in the replay spec. The missed detection comes from deferral,
|
||||
rather than a limit in Joern's capability.
|
||||
|
||||
## Anchors (confirm in increment 1)
|
||||
|
||||
- `src/openultrasast/rank.py` — the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
|
||||
- `src/openultrasast/mapping.py` — builds the `selected`/`deferred` scope + the budget allocation that defers on
|
||||
- `src/openultrasast/rank.py` is the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
|
||||
- `src/openultrasast/mapping.py` builds the `selected`/`deferred` scope + the budget allocation that defers on
|
||||
`deadline_exhausted`.
|
||||
- `src/openultrasast/push/policy.py`, `push/runner.py` — consume the scope; change_sites arrive via the push input.
|
||||
- `benchmarks/unseen/replay_entry.py` — carries `changed_paths` / `declaration_paths` in the spec.
|
||||
- `src/openultrasast/push/policy.py` and `push/runner.py` consume the scope; change_sites arrive via the push input.
|
||||
- `benchmarks/unseen/replay_entry.py` carries `changed_paths` / `declaration_paths` in the spec.
|
||||
|
||||
## Increments (progress tracking)
|
||||
|
||||
- [x] **0. Cost split confirmed (2026-10-11, timed live replay).** `build_seconds=66.69` (CPG build is cheap),
|
||||
`query_seconds=0.0` (no query ever executed), total 303 s (hit the deadline), **4,095** candidate questions all
|
||||
`analysis_not_started=deadline_exhausted`, `selected: []`, `regions_scanned=0`. So of the 303 s, ~67 s built the
|
||||
CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change —
|
||||
the budget was spent before a single query ran. The CPG build is NOT the bottleneck; the over-broad candidate set
|
||||
is. Change-scoping is confirmed as the right lever, and because per-candidate prep includes the line-correspondence
|
||||
work, fewer candidates also cuts that cost (the two fixes compound).
|
||||
CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change.
|
||||
The budget was spent before a single query ran. The over-broad candidate set is the bottleneck, rather than the CPG build.
|
||||
This confirms change-scoping as the fix. Preparing each candidate includes line-correspondence work,
|
||||
so fewer candidates also reduce that cost.
|
||||
- [x] **1. Located (2026-10-11).** The over-enumeration is upstream of the scan, not inside it:
|
||||
- `push/runner.py:345-348` — `head_regions = discovery(tip.root, …)` discovers regions across the **whole head
|
||||
- `push/runner.py:345-348`, `head_regions = discovery(tip.root, …)` discovers regions across the **whole head
|
||||
tree** (the ~4,095), independent of the change.
|
||||
- `push/runner.py:397-407` — those regions are passed wholesale to `scan_repository(..., change_context=context)`;
|
||||
- `push/runner.py:397-407`, those regions are passed wholesale to `scan_repository(..., change_context=context)`;
|
||||
`change_context` is threaded in but **never scopes the region set**.
|
||||
- `model/scan.py:167` and `:364` — `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all
|
||||
- `model/scan.py:167` and `:364`, `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all
|
||||
regions × families; with `order_by_evidence=True` (`runner.py:288/403`) the scan evidence-tiers **every** pair
|
||||
before analysis — that pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`.
|
||||
before analysis. That pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`.
|
||||
- Primitive already present: `model/regions.py:279` computes the changed-path set from the context;
|
||||
`compare_targeted_base` (`runner.py:411`) is a precedent for change-targeted work.
|
||||
- **Fix locus (increment 2):** restrict the *analyzed* region set to the change's declarations + dataflow
|
||||
- For increment 2, restrict the *analyzed* region set to the change's declarations + dataflow
|
||||
neighbourhood (filter `head_regions` by `context.changed_paths` before the scan, or scope the enumeration inside
|
||||
`scan_repository`). Keep the **CPG whole-repo** so taint queries can still trace sources from unchanged code into
|
||||
the changed regions — only the question set is change-scoped.
|
||||
the changed regions. Only the question set is change-scoped.
|
||||
|
||||
**Correction (2026-10-11), after a reverted first attempt — two things the naive filter got wrong:**
|
||||
Correction (2026-10-11), after reverting the first attempt: the filter got two things wrong.
|
||||
- **The engine already reaches into unchanged code by design, and it is tested.** A changed `source` that reaches
|
||||
an unchanged `sink` must still select the unchanged sink's question
|
||||
(`tests/test_model_change_context.py::test_changed_dependency_reaches_unchanged_ranked_question`;
|
||||
`test_change_does_not_replace_ranker_selection` pins that change context must NOT alter the ranker's selection).
|
||||
`_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths` — which locations to itemise vs
|
||||
summarise — NOT to shrink the tier population. A "changed files only" scope deletes the reach-into-unchanged
|
||||
`test_change_does_not_replace_ranker_selection` pins that change context must not alter the ranker's selection).
|
||||
`_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths`. This controls which locations to itemise vs
|
||||
summarise, rather than shrinking the tier population. A "changed files only" scope deletes the reach-into-unchanged
|
||||
behaviour. Any scope must keep changed regions **plus the regions they reach** (the affected/`relationships`
|
||||
neighbourhood), not changed files alone.
|
||||
- **The 236 s is now split (2026-10-11, saved artifacts).** `total=291s build=52s query=0s arbitrate=238.75s`,
|
||||
and `query_seconds_by_kind["evidence"]=238.54s` — the evidence-tier pass is **99.9 %** of the arbitrate phase;
|
||||
and `query_seconds_by_kind["evidence"]=238.54s`, the evidence-tier pass is **99.9 %** of the arbitrate phase;
|
||||
`affected_context` + `_scope_work` are **0.21 s**. The pass tiered 1,015 pairs (`tier_counts 0:1009, 1:6`) at
|
||||
**~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget — `selected=0`,
|
||||
**~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget, `selected=0`,
|
||||
`deferred=1365` all `deadline_exhausted`. Secondary anomaly: the design assumes ~2.5 ms/pair
|
||||
("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is pathologically slow on a cold large
|
||||
CPG — a separate optimisation from cutting the pair count.
|
||||
- **So increment 2's single target is the evidence pass** (`scan.py:338-359`, `_evidence_pass`). The fork is HOW
|
||||
to cut it without deleting reach-into-unchanged:
|
||||
- (i) **Scope + bound:** order the evidence candidates change-first and bound the pass to a fraction of the
|
||||
("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is unusually slow on a cold large
|
||||
CPG. Fixing that is separate from cutting the pair count.
|
||||
- Increment 2 targets the evidence pass (`scan.py:338-359`, `_evidence_pass`). The options must
|
||||
reduce its cost while preserving flows into unchanged code:
|
||||
- (i) Scope + bound: order the evidence candidates change-first and bound the pass to a fraction of the
|
||||
budget so analysis queries still run; reached unchanged regions are tiered only while budget lasts.
|
||||
- (ii) **Neighbourhood pass:** tier the changed regions, read the reached regions from the context rows the
|
||||
pass already returns, tier those in a second small pass — exact reach-into-unchanged, more code.
|
||||
- (iii) **Per-pair speedup:** attack the ~235 ms/pair (warm server / batching) rather than the pair count —
|
||||
helps every scan but is a deeper engine change and does not by itself bound a huge repo.
|
||||
- (ii) Neighbourhood pass: tier the changed regions, read the reached regions from the context rows the
|
||||
pass already returns, then tier those in a second small pass. This preserves exact reach-into-unchanged behaviour but needs more code.
|
||||
- (iii) Per-pair speedup: reduce the ~235 ms/pair (warm server / batching) rather than the pair count.
|
||||
This helps every scan but needs a deeper engine change and does not by itself bound work on a huge repo.
|
||||
|
||||
**CORRECTION (2026-10-11, after implementing (i) + four live re-verifies — the cost model was wrong).**
|
||||
**CORRECTION (2026-10-11, after implementing (i) + four live re-verifies, the cost model was wrong).**
|
||||
Scope + bound was built and tested (change-first ordering, adaptive budget-fitted portioning, `_context_pass`
|
||||
scoped to changed files, honest `evidence_pass_bounded` degradation; branch `engine-change-scoped-evidence-pass`).
|
||||
It correctly cut the tiered pair count — but `query_seconds` stayed **0** every time. The live numbers show why:
|
||||
It cut the tiered pair count, but `query_seconds` stayed **0** every time. The live numbers show why:
|
||||
|
||||
| pairs tiered | run_batch calls | evidence s | build s |
|
||||
|---|---|---|---|
|
||||
@@ -99,53 +99,53 @@ detection is being lost, not in Joern's capability.
|
||||
Evidence time is **flat regardless of pair count** → the cost is **per-*call* fixed** (~150–220 s: loading the CPG
|
||||
into the query engine + the census the evidenceOnly batch computes), ~85 ms/pair on top. Pair-count scoping
|
||||
therefore cannot free budget, and portioning re-pays the fixed cost per call (so (i) can be counterproductive).
|
||||
Build is also large and swings 52→127 s — two engines thrashing the 7 GB host (the record-07 baseline shares it).
|
||||
The real levers are now (iii)-shaped, not (i): **warm/persistent query engine or a cached census** to kill the
|
||||
~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s; plus a **clean (uncontended)
|
||||
measurement**, because contention alone may be inflating build + fixed cost 2–3×. Scope+bound is NOT merged.
|
||||
Build time also varies from 52→127 s. Two engines compete for the 7 GB host, which the record-07 baseline shares.
|
||||
The fixes now resemble (iii), rather than (i): a **warm/persistent query engine or a cached census** to remove the
|
||||
~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s. A **clean (uncontended)
|
||||
measurement** is also needed because contention may inflate build + fixed cost 2–3×. Scope+bound is not merged.
|
||||
|
||||
**UNCONTENDED measurement (2026-10-11, baseline paused via SIGSTOP + container kill, then resumed).** Shipped
|
||||
engine, one evidence call over 1,015 pairs: `build=48 s`, `evidence=243.79 s`, `query=0`. The evidence pass is
|
||||
**the same ~244 s uncontended as ~238 s contended** → the cost is **INTRINSIC, not contention**; contention only
|
||||
**the same ~244 s uncontended as ~238 s contended** → the cost is **intrinsic, not contention**; contention only
|
||||
inflated `build` (48 → 127 s) and added run-to-run noise. The model is `evidence ≈ 150 s fixed (CPG load into the
|
||||
query engine + the census the evidenceOnly batch computes) + ~93 ms/pair` (244 = 150 + 1015×0.093). Consequences:
|
||||
- **~200 s of fixed overhead** (`build` + evidence-load) precedes any analysis, so a 300 s budget leaves only
|
||||
~100 s for everything else. This is why `query=0` even on a clean host.
|
||||
- **Change-scoping still helps, but only by cutting the per-pair tail** (~1,015×93 ms ≈ 95 s): one evidence call
|
||||
over ~30 changed pairs is ~150 s + 3 s instead of 244 s, which would free ~90 s — possibly enough for analysis,
|
||||
but marginal.
|
||||
over ~30 changed pairs is ~150 s + 3 s instead of 244 s. This would free ~90 s, which might leave enough time
|
||||
for analysis, but only just.
|
||||
- **It must be ONE call, not portioned.** The ~150 s is a first-call load, so the adaptive probe attributed it to
|
||||
its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4 — the implemented
|
||||
scope+bound is doubly wrong for this cost model. The correct shape: change-first candidates, count-capped, ONE
|
||||
its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4, the implemented
|
||||
scope+bound has both problems under this cost model. The correct shape: change-first candidates, count-capped, ONE
|
||||
evidence call, `deadline=None`.
|
||||
- **Pre-push viability:** the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a
|
||||
- Pre-push viability: the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a
|
||||
repo this size without a warm/persistent session or a cached census. A larger budget (e.g. 600 s) makes the
|
||||
benchmark arm run but is not a realistic pre-push deadline. The lever that matters for the product is killing
|
||||
the ~150 s fixed cost, not scoping pairs.
|
||||
- [ ] **2. Seed analysis from the change.** Prioritize (or restrict first-pass to) the changed declarations plus
|
||||
their call/data neighborhood over repo-wide static rank, within budget.
|
||||
- [ ] **3. Soundness.** Keep cross-file dataflow INTO the changed function (don't scope to literal changed lines);
|
||||
- [ ] **3. Soundness.** Keep cross-file dataflow into the changed function (don't scope to literal changed lines);
|
||||
add tests; keep existing push tests green.
|
||||
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now RUN and produce a finding on
|
||||
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now run and produce a finding on
|
||||
`install_packages` (vs `deferred`)? Re-run metrics-plugin.
|
||||
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate) — needs the
|
||||
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate), needs the
|
||||
kube lane (blocked on the AX completion path) for scale.
|
||||
|
||||
## Dependencies / notes
|
||||
|
||||
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; the win
|
||||
here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
|
||||
- Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine
|
||||
changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here.
|
||||
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; this work
|
||||
moves findings from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
|
||||
- Implement directly and verify on the live instrument (a fake-client/unit pass is not evidence the live engine
|
||||
changed); the kube-lane and store endpoints are infrastructure owned by the operator.
|
||||
|
||||
## Efficiency barrier analysis (post-record-07 priority)
|
||||
|
||||
**What the record-07 baseline actually measured (verified on the records): regex only.** The default and
|
||||
long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0` — **no LLM is
|
||||
called** — and the engine never completed (`engine_covered=0`, the ~200 s wall). So the measured `catch≈2.5%` is
|
||||
the **quick pattern-rule tier alone**: essentially grep-with-structure. This is the FLOOR, not the product, and it
|
||||
confirms the maintainer's point — pattern rules alone will never be enough, or `grep` would do. The product's
|
||||
detection lives in two tiers that did NOT run:
|
||||
The records confirm that the record-07 baseline measured regex only. The default and
|
||||
long_deadline replay arms both hardcode `model="disabled"`, `max_model_calls=0`, `cost_usd=0`. No LLM is
|
||||
called, and the engine never completed (`engine_covered=0`, the ~200 s wall). The measured `catch≈2.5%` covers
|
||||
only the quick pattern-rule tier, a structured pattern search. This measures the floor, rather than the product.
|
||||
It confirms the maintainer's point that pattern rules alone are insufficient; otherwise `grep` would do.
|
||||
The product's two detection tiers did not run:
|
||||
|
||||
| Tier | Mechanism | Measured cost (uncontended, mid-size repo) | Ran? | Catch |
|
||||
|---|---|---|---|---|
|
||||
@@ -153,20 +153,20 @@ detection lives in two tiers that did NOT run:
|
||||
| 2 engine (Joern) | CPG build + taint/dataflow | build 48 s + evidence (150 s fixed + 93 ms/pair) + analysis (unmeasured) | no (walled) | unknown |
|
||||
| 3 LLM adjudication | model judgment / search-with-proof (hunter, fusion, decision engine) | model latency + $/call (unmeasured in the push path) | no (disabled) | unknown |
|
||||
|
||||
**The barrier:** a releasable net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window —
|
||||
target **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis is a budget ledger per tier against
|
||||
that window, and the measurements each needs:
|
||||
To be releasable, the net needs tier 2 and/or tier 3 to run and catch within a realistic pre-push window.
|
||||
The target is **≤60–120 s** (developer patience), hard ceiling 300 s. The analysis records each tier's budget against
|
||||
that window and identifies the measurements needed:
|
||||
|
||||
- **Engine barrier (tier 2).** Fixed floor `build(48) + evidence_fixed(~150) ≈ 198 s` precedes any dataflow answer.
|
||||
Measure (clean host): **E1** split the ~150 s into JVM-start / graph-load / census; **E2** does a warm session
|
||||
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the decisive architectural
|
||||
with the graph **preloaded at build** eliminate `evidence_fixed` on the first query (the architectural
|
||||
test)?; **E3** analysis (`reachableByFlows`) per-pair cost + warm reuse; **E4** build cost vs repo size.
|
||||
Levers in priority order: warm session + graph preload → cached census → change-scope the per-pair tail.
|
||||
- **LLM barrier (tier 3).** Unmeasured in the push path. Measure: **L1** model round-trip latency per call
|
||||
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall LIFT of tier 3 over tiers
|
||||
1+2 on the pool — does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
|
||||
- **Deliverable:** a **barrier table** — for budget ∈ {60, 120, 300, 600} s × each tier combination, does it FIT
|
||||
and what is the MEASURED recall/false-alarm on the pool — identifying the minimal viable config for a releasable
|
||||
(DeepSeek) and calls needed per push for the decision/proof stage; **L2** the recall lift of tier 3 over tiers
|
||||
1+2 on the pool, does the LLM catch what regex and dataflow miss, and at what $ and latency per push?
|
||||
- Produce a **barrier table** for budget ∈ {60, 120, 300, 600} s × each tier combination. Record whether it fits
|
||||
and its measured recall/false-alarm on the pool. Use this to identify the minimal viable config for a releasable
|
||||
net. 600 s unblocks the benchmark arm but is not a pre-push deadline; the product target is tiers 2+3 under ~120 s.
|
||||
|
||||
Sequence: finish G0 (record 07) → **E1/E2 first** (they decide whether the warm session alone crosses the engine
|
||||
@@ -176,8 +176,8 @@ lever.
|
||||
|
||||
## E1/E2 RESULTS (measured 2026-10-12, CLEAN host, bedrock-agentcore, 3-file change, 300 s budget, head side)
|
||||
|
||||
Four clean-host replays settle the engine barrier and **correct the earlier "~200 s graph-load wall"**, which was
|
||||
conflated with host contention from the running baseline replay:
|
||||
Four clean-host replays **correct the earlier "~200 s graph-load wall"** diagnosis. It had confused graph-load
|
||||
cost with host contention from the running baseline replay:
|
||||
|
||||
| Run | build | evidence (taint batch) | dominance | query_s | selected | deferred |
|
||||
|---|---|---|---|---|---|---|
|
||||
@@ -186,40 +186,40 @@ conflated with host contention from the running baseline replay:
|
||||
| warm session `SESSION=1` (clean) | 32.9 | 228.6 | 25.9 | 25.9 | **356** | 1009 |
|
||||
| session + graph preload (clean) | 30.4 | 223.2 | 32.5 | 32.5 | **356** | 1009 |
|
||||
|
||||
**Findings (each traced to a measured cause, not assumed):**
|
||||
The measurements show:
|
||||
|
||||
1. **The engine arm was walled by CONTENTION, not an intrinsic floor.** The prior `query=0 / selected=0 / all
|
||||
1. Host contention prevented the engine arm from running. The prior `query=0 / selected=0 / all
|
||||
1365 deferred` was measured while the slice-1 baseline replay contended for the 7 GB host. On a clean host the
|
||||
**shipped (disposable) engine already runs dataflow**: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**,
|
||||
shipped (disposable) engine already runs dataflow: `query_seconds` 0 → ~35 s, `selected` 0 → **356 regions**,
|
||||
dominance flow queries complete. This is the first time the engine arm produced analysis on this change.
|
||||
2. **The evidence/taint batch (~225 s) is INVARIANT to session and to graph preload** (225.1 / 228.6 / 223.2 across
|
||||
2. **The evidence/taint batch (~225 s) is invariant to session and to graph preload** (225.1 / 228.6 / 223.2 across
|
||||
the three clean runs). Warm session and preload-at-build do **not** reduce it. So the dominant cost is intrinsic
|
||||
to the taint batch, not graph I/O.
|
||||
3. **Graph-load is cheap — it is NOT the wall.** The `dominance` query is a full cold query *including*
|
||||
3. Graph-load is cheap and does not account for the delay. The `dominance` query is a full cold query *including*
|
||||
`importCpg`, and it costs only ~35 s. So `importCpg` is tens of seconds, not ~150 s. **This corrects the earlier
|
||||
note** that attributed the ~150 s fixed to "first query JVM-start + graph-load + census." The ~160 s fixed lives
|
||||
inside `taint.sc`'s own per-batch flow work (seed/sink memo fill over the distinct `(family, file)` combinations
|
||||
it walks), paid lazily as the first pairs process — independent of the raw pair count.
|
||||
it walks). It is paid lazily as the first pairs are processed, independent of the raw pair count.
|
||||
4. **The warm session is not the lever.** It starts fine on a clean host (the contended `session_startup_timeout`
|
||||
was contention-only), but gives ~no benefit here: subsequent queries (dominance) are already cheap disposable,
|
||||
and the evidence pass is a single batch call, so there is nothing for the session to amortise.
|
||||
5. **The barrier, restated:** `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer
|
||||
as `deadline_exhausted`; only 356 are analysed. The lever is **reducing the taint batch's work** — change-scope
|
||||
the pair/family set so the batch fits budget — NOT session, preload, or a bigger JVM. The parked branch
|
||||
5. The barrier, restated: `build(~30) + evidence(~225) ≈ 255 s` of a 300 s budget, so 1009/1365 regions defer
|
||||
as `deadline_exhausted`; only 356 are analysed. The fix is to **reduce the taint batch's work** by scoping
|
||||
the pair/family set to the change so the batch fits the budget. Session reuse, preload, or a bigger JVM will not do this. The parked branch
|
||||
(`engine-change-scoped-evidence-pass`) is therefore **relevant after all, on a clean host**, provided it scopes
|
||||
the family/file set `taint.sc` walks (not merely the pair list); its adaptive portioning still needs replacing
|
||||
with a fixed cap (per-call-fixed cost, not per-pair).
|
||||
6. **Side note:** the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run — a separate
|
||||
6. Side note: the `base`-side CPG build failed (`cpg_build_failed`, 0.9 s) in the preload run, a separate
|
||||
intermittent base-revision build issue; the `head` side built and analysed normally. Track separately.
|
||||
|
||||
**What E1/E2 did NOT answer (the real G2 question):** on a clean host the engine selects 356 regions — *does it
|
||||
CATCH vulns regex misses?* That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the
|
||||
engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the decisive
|
||||
E1/E2 did not answer the G2 question: does the engine catch vulnerabilities regex misses when it selects 356 regions
|
||||
on a clean host? That needs an **engine-on unseen pilot** (a handful of pool repos replayed with the
|
||||
engine arm enabled on a clean host) scored against record 07's regex-only floor (catch 3/92). That is the
|
||||
next experiment, ahead of E3/E4/L1/L2.
|
||||
|
||||
## ENGINE-ON PILOT RESULTS (measured 2026-10-12, CLEAN host, 5 public pool repos, vuln change each, 300 s)
|
||||
|
||||
Five public repos picked for flow-shaped families and small size (engine's best chance to finish and catch):
|
||||
Five public repos were picked for flow-shaped families and small size, giving the engine its best chance to finish and catch:
|
||||
|
||||
| repo | fam | size | REGEX on vuln | ENGINE finding | cov | vuln region ranked | best taint | selected/deferred | wall |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
@@ -229,70 +229,70 @@ Five public repos picked for flow-shaped families and small size (engine's best
|
||||
| aws/bedrock-agentcore | injection | 3f | ✗ | **none** | incomplete | yes | — | 356 / 1009 | 305 s |
|
||||
| miguelgrinberg/python-socketio | deser | 10f | **✓ (n=2)** | **none** | incomplete | yes | **tier1 score2.0 sink found** | 136 / **0** | 257 s |
|
||||
|
||||
**Result: regex 1/5, engine 0/5.** On a clean host, with budget to spare on most, the shipped engine **added no
|
||||
Result: regex 1/5, engine 0/5. On a clean host, with budget to spare on most, the shipped engine **added no
|
||||
catches over the regex floor.**
|
||||
|
||||
**Why (measured, traced — CORRECTED after inspecting `head.findings`, not ranker evidence):**
|
||||
1. **The budget hypothesis is REFUTED.** banks, graphiti and socketio **finished their region queue** (deferred=0)
|
||||
The diagnosis was corrected after inspecting `head.findings` rather than ranker evidence:
|
||||
1. The budget hypothesis is refuted. banks, graphiti and socketio **finished their region queue** (deferred=0)
|
||||
and still emitted no finding. Change-scoping to fit budget would not have changed the outcome.
|
||||
2. **The engine produced ZERO vuln-family findings (path/injection/deser) on all 5.** Actual `head.findings`:
|
||||
2. The engine produced no vuln-family findings (path/injection/deser) on any of the 5. Actual `head.findings`:
|
||||
banks 1 (config_secrets), socketio 2 (config_secrets), launch-editor / graphiti / bedrock **0 findings each**.
|
||||
The dataflow/taint tier traced **no source→sink flow** for the vuln families (`taint_evidence_not_applicable`;
|
||||
e.g. banks found the `path.open` sink but **no source reaches it**). **Correction:** the tier=1 / score=2.0
|
||||
"sink" seen in the ranker's `normalized_evidence` is the RANKER's static sink candidate, NOT a confirmed taint
|
||||
finding — an earlier draft of this section misread it as detection partially working. It was not.
|
||||
3. **`ambiguous_line_correspondence` is a downstream RED HERRING for these vulns.** It fires on all 5, but there
|
||||
e.g. banks found the `path.open` sink but **no source reaches it**). Correction: the tier=1 / score=2.0
|
||||
"sink" seen in the ranker's `normalized_evidence` is the ranker's static sink candidate, not a confirmed taint
|
||||
finding. An earlier draft of this section misread it as detection partially working. It was not.
|
||||
3. `ambiguous_line_correspondence` does not explain these missed vulnerabilities. It fires on all 5, but there
|
||||
are **no vuln-family findings to anchor**, so a region-level witness fallback (the proposed "Option A") would
|
||||
change nothing here. Witness/anchoring only touched the config_secrets findings that did appear.
|
||||
4. **The LLM adjudication tier was DISABLED** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The
|
||||
4. **The LLM adjudication tier was disabled** (`model_calls=0`, `cost_usd=0`, `model_assistance={}` on all 5). The
|
||||
rungs reached: `suspicion=0, model_corroborated=0, model_entailed=0–2, execution_confirmed=0`. The tier designed
|
||||
to confirm a vuln the static dataflow cannot trace **never ran**.
|
||||
5. **Ranking also under-promotes the vuln region** (tier 0 for most families; banks path family stuck low). So even
|
||||
where a sink exists, the region is not analysed deeply enough to attempt a flow.
|
||||
6. **Measurement caveat:** the pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe
|
||||
6. The pool's SZZ last-writer change definition yields large/moved diffs; a cleaner probe
|
||||
uses tight vuln-introducing diffs. Secondary to (2)/(4).
|
||||
|
||||
**Reprioritised engine levers (CORRECTED — anchoring is NOT the lever):**
|
||||
- **#1 the LLM adjudication tier (disabled in the push path) — now the primary lever.** regex+dataflow = 1/5 with
|
||||
the model OFF; the static tiers produce no vuln-family findings on these shapes. The product thesis
|
||||
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot is the empirical case to
|
||||
turn it on: **L1** model latency + calls/push, **L2** recall LIFT on the same 5 (and beyond). Costs money — stage
|
||||
it (5–10 sizing → ~10–20 pilot → full only on lift).
|
||||
Reprioritised engine levers (corrected, anchoring is not the lever):
|
||||
- #1: the LLM adjudication tier (disabled in the push path) has the highest priority. regex+dataflow = 1/5 with
|
||||
the model OFF; the static tiers produce no vuln-family findings on these shapes. The hypothesis
|
||||
([[search-with-proof]]) is that the LLM tier catches what regex+dataflow miss. The pilot supports
|
||||
testing it: **L1** model latency + calls/push, **L2** recall lift on the same 5 (and beyond). These calls cost money, so stage
|
||||
them (5–10 sizing → ~10–20 pilot → full only on lift).
|
||||
- **#2 taint source/sink coverage + region ranking** so the dataflow recognises these vuln sources and promotes the
|
||||
vuln region (banks path: sink found, source unrecognised; graphiti injection: score 0).
|
||||
- **#3 change-anchoring robustness** — real, but only matters AFTER the static/LLM tiers produce vuln-family
|
||||
- #3 change-anchoring robustness matters only after the static/LLM tiers produce vuln-family
|
||||
findings to anchor. Deprioritised until (1)/(2) yield findings.
|
||||
|
||||
**Release read:** the pre-push net is NOT releasable on detection. Clean host, generous budget, LLM off: regex 1/5,
|
||||
engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the decisive untested lever
|
||||
is the **LLM adjudication tier**. Next: an **LLM-tier pilot (L1/L2)** on these 5 repos — does enabling the model
|
||||
produce vuln-family findings regex+dataflow miss, and at what $/latency.
|
||||
Detection does not meet the pre-push net's release requirements. Clean host, generous budget, LLM off: regex 1/5,
|
||||
engine 0/5. The static tiers (regex + Joern dataflow) do not detect these unseen vulns; the untested lever
|
||||
is the **LLM adjudication tier**. Next, run an **LLM-tier pilot (L1/L2)** on these 5 repos to test whether the model
|
||||
produces vuln-family findings regex+dataflow miss, and measure its $/latency.
|
||||
|
||||
## ROOT CAUSE of engine 0/5 + the "#2 guided by #1" workstream (2026-10-12)
|
||||
|
||||
**Why the taint tier traced no flow (verified in code + evidence):** the pilot repos are **libraries**, whose
|
||||
Code and evidence show why the taint tier traced no flow. The pilot repos are **libraries**, whose
|
||||
untrusted input arrives as **public-API function parameters**, not HTTP requests. The Python taint **source**
|
||||
vocabulary (`ruleset/semantic/python.toml`) is entirely web/CLI: Flask/Django `request.*`, `sys.argv`, `sys.stdin`,
|
||||
`input(`. There is no source for a library's own API parameter. The engine *does* have a parameter-as-source mode
|
||||
(`_parameters_are_input`, `scan.py`), but it fires **only for recognised entry points** (`_is_entry_point` and
|
||||
`entry_kind != "function"`) — the pilot vuln regions carry `entry:false`, so their parameters are **deliberately
|
||||
`entry_kind != "function"`). The pilot vuln regions carry `entry:false`, so their parameters are **deliberately
|
||||
not** sources. That restriction is intentional: treating every function parameter as a source caused a flood of
|
||||
false SQL-injections on WordPress/PMPro (documented in the `_parameters_are_input` docstring). So **the static
|
||||
engine cannot safely widen its sources to library APIs without reintroducing that false-alarm explosion** —
|
||||
`banks` path: sink `path.open` found, `source_local/near=false`; `graphiti`/`bedrock` injection: no source.
|
||||
false SQL-injections on WordPress/PMPro (documented in the `_parameters_are_input` docstring). The static
|
||||
engine cannot safely widen its sources to library APIs without reintroducing those false alarms.
|
||||
For `banks` path: sink `path.open` found, `source_local/near=false`; `graphiti`/`bedrock` injection: no source.
|
||||
|
||||
**This is the precise seam for the maintainer's "#2 guided by #1" directive.** The static taint cannot *safely*
|
||||
The maintainer's "#2 guided by #1" directive addresses this gap. Static taint cannot *safely*
|
||||
decide which library parameters are trust boundaries; the **LLM can** reason about which public-API surface is
|
||||
attacker-reachable (intent, not pattern). So:
|
||||
- **#1 (LLM) guides:** on a changed library region the engine ranks but cannot source, the LLM adjudicates whether a
|
||||
- #1 (LLM) guides: on a changed library region the engine ranks but cannot source, the LLM adjudicates whether a
|
||||
parameter is an untrusted trust boundary and names the source→sink it believes exists (search-with-proof).
|
||||
- **#2 (taint) traces + proves:** the engine then runs the flow from the LLM-named source to the known sink,
|
||||
confirming (or refuting) with a traced path — a proof, not the LLM's say-so. Net: catch without the static FP
|
||||
blowup, and every catch carries a dataflow witness.
|
||||
- **Feedback:** where the LLM repeatedly names the same source shape (e.g. a registry-key → path family), promote it
|
||||
- #2 (taint) traces + proves: the engine then runs the flow from the LLM-named source to the known sink,
|
||||
confirming or refuting it with a traced path. The trace provides proof beyond the LLM's judgment. This would avoid the static FP
|
||||
increase while giving every catch a dataflow witness.
|
||||
- Where the LLM repeatedly names the same source shape (e.g. a registry-key → path family), promote it
|
||||
to a static fact in `python.toml` guarded by the entry/shape conditions that keep WordPress quiet.
|
||||
|
||||
**Proposed first increment (pending budget confirmation — LLM calls cost money):** wire the disabled LLM tier on the
|
||||
Proposed first increment (pending budget confirmation, LLM calls cost money): wire the disabled LLM tier on the
|
||||
5 pilot regions in *LLM-guided-source* mode and measure: (a) does it name the right trust-boundary source per vuln,
|
||||
(b) does the taint engine then trace source→sink and emit a vuln-family finding, (c) catch lift over regex (1/5) and
|
||||
false-alarm cost on the ordinary changes. Stage it (5–10 sizing → ~10–20 → full on lift). This makes "#2 guided by
|
||||
@@ -303,79 +303,78 @@ yields findings.
|
||||
|
||||
Sized the LLM tier on banks (the clearest pilot miss) with the project's own DeepSeek client (`deepseek-flash`).
|
||||
|
||||
**Infra + cost (sizing answer):** DeepSeek reachable, model id `deepseek-flash` valid. **~$0.00016–0.00025 per
|
||||
call, ~1.1 s latency.** Sub-cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback
|
||||
(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param — a client fix (don't send
|
||||
DeepSeek is reachable, and model id `deepseek-flash` is valid. **~$0.00016–0.00025 per
|
||||
call, ~1.1 s latency.** This costs less than a cent per push even at 8 candidates × a few regions. The GLM/abliteration fallback
|
||||
(`ABLITERATION_*`) currently errors `HTTP 422` on the `thinking` request param. A client fix (don't send
|
||||
`thinking` to that endpoint) is needed before it can be used; DeepSeek works today so it is not blocking.
|
||||
|
||||
**"2 guided by 1" validated as a mechanism — the LLM gives the correct verdict WHEN given the real sink + source
|
||||
context:**
|
||||
- Asked about `_load` (the pool's SZZ-labelled site, 135–141): `vulnerable=false` — *correct*, `_load` reconstructs
|
||||
"2 guided by 1" works in this case: the LLM gives the correct verdict when given the real sink + source
|
||||
context:
|
||||
- For `_load` (the pool's SZZ-labelled site, 135–141), the model returned `vulnerable=false`. This is correct: `_load` reconstructs
|
||||
paths from its own index, not the sink.
|
||||
- Asked about the REAL sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the
|
||||
context that `prompt.name` can be request-derived: **`vulnerable=true`** — *"prompt.name is joined into a
|
||||
filesystem path and written without sanitization… path traversal or arbitrary file write."* A correct catch.
|
||||
- Asked about the real sink `from_prompt_path`/`set()` line 44 (`path / f"{prompt.name}…"; write_text`) with the
|
||||
context that `prompt.name` can be request-derived: **`vulnerable=true`**, *"prompt.name is joined into a
|
||||
filesystem path and written without sanitization… path traversal or arbitrary file write."* This is a correct catch.
|
||||
|
||||
**Two blockers to realising it in the live pipeline (both now concrete):**
|
||||
Two blockers to realising it in the live pipeline (both now concrete):
|
||||
1. **Context, not single-shot.** The model only catches with the real sink + caller/source context. A single-shot
|
||||
question on the ranker's region (which pointed at `_load`, and at the benign local-glob `_scan`) does not. This
|
||||
requires the **agentic search-with-proof hunter** (explores callers/entry points to establish the trust
|
||||
boundary), not the single CANDIDATE_QUESTION path. That is more calls → the real cost driver to size next.
|
||||
2. **Corpus-label soundness.** The pool's banks label is **mislocated**: `head_lines=[[30],[104],[138–141]]`,
|
||||
`function=_load` — the real sink is line **44**, outside head_lines±10 and a different function. So (a) even a
|
||||
correct finding at line 44 would NOT score as a catch under `location_match`, and (b) the sink pre-exists in base
|
||||
(the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet** — the pool even
|
||||
boundary), not the single CANDIDATE_QUESTION path. That needs more calls → their cost is the next measurement.
|
||||
2. **Corpus-label soundness.** The pool's banks label is mislocated: `head_lines=[[30],[104],[138–141]]`,
|
||||
`function=_load`. The real sink is line **44**, outside head_lines±10 and a different function. So (a) even a
|
||||
correct finding at line 44 would not score as a catch under `location_match`, and (b) the sink pre-exists in base
|
||||
(the change did not touch it), so a change-scoped pre-push hook **correctly stays quiet**. The pool even
|
||||
annotates this bias ("delta hook can correctly stay quiet"). **Part of the pilot's engine 0/5 is this SZZ
|
||||
mislocation, not a detection failure.**
|
||||
|
||||
**Revised next steps (budget approved, stay staged):**
|
||||
The budget is approved. Keep the revised next steps staged:
|
||||
- **Validate the 5 pilot labels first** (cheap, LLM-assisted corpus audit): for each, is the real sink inside the
|
||||
changed lines, and is it change-introduced vs pre-existing-in-base? Only sound, change-located vulns are fair
|
||||
targets. This separates true detection gaps from SZZ artifacts before spending on detection runs.
|
||||
- **Then run the AGENTIC hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
|
||||
- **Then run the agentic hunter** (search-with-proof) on the sound targets and size its per-vuln call count/cost;
|
||||
measure catch + false-alarm (on ordinary changes) vs regex 1/5. The single-shot path is insufficient; the hunter
|
||||
is the real "#1".
|
||||
- Feed confirmed source shapes (e.g. library param → path join) back as guarded static facts (#2).
|
||||
|
||||
## Prompt evolution is a CORE lever for the LLM arm (maintainer directive 2026-10-12)
|
||||
|
||||
**"What is key is that we evolve the prompts as well"** — and **"GEPA or comparable; this is an evolving field."**
|
||||
The bedrock hunter result makes this concrete: the agentic hunter ran (64 model calls, $0.0076) and reasoned
|
||||
carefully, but with its **current prompts** it flagged adjacent concerns (network-PUBLIC default, download_file)
|
||||
and never named the actual `install_packages` command injection — a *prompting/targeting* miss, not a cost or
|
||||
capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence
|
||||
outputs, evolves the prompt, and retries — built on **DSPy** (its GEPA / MIPRO optimisers), kept **pluggable**
|
||||
behind an interface because the field moves fast (GEPA, MIPRO, TextGrad, successors). This aligns with
|
||||
[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to BOTH LLM
|
||||
uses here — the search-with-proof detector and the corpus-label auditor.
|
||||
**"What is key is that we evolve the prompts as well"**, and **"GEPA or comparable; this is an evolving field."**
|
||||
The bedrock hunter ran (64 model calls, $0.0076), but its **current prompts** led it to flag adjacent concerns
|
||||
(network-PUBLIC default, download_file). It never named the `install_packages` command injection. This was a
|
||||
*prompting/targeting* miss, rather than a cost or capability limit. So the LLM arm needs a **prompt-optimisation loop** that reflects on misses/low-confidence
|
||||
outputs, updates the prompt, and retries. Build it on **DSPy** (its GEPA / MIPRO optimisers). Keep it **pluggable**
|
||||
behind an interface because the field changes quickly (GEPA, MIPRO, TextGrad, successors). This aligns with
|
||||
[[learned-decisions-not-hand-tuning]] (learned decisions + local memory, not hand rules). It applies to both LLM
|
||||
uses here: the search-with-proof detector and the corpus-label auditor.
|
||||
|
||||
**The optimisation objective is to learn WHAT HINTS/ANNOTATIONS the model needs** to reach the real sink — not
|
||||
just reword the prompt. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package
|
||||
The optimisation must learn which hints/annotations the model needs to reach the real sink. Rewording the
|
||||
prompt alone is insufficient. bedrock needed the hint "`install_packages` runs `pip install` with caller-supplied package
|
||||
names → command injection"; banks needed "`prompt.name` is request-derived and joined into a filesystem path →
|
||||
traversal". DSPy optimises which evidence/hints to feed the signature so the model targets the true sink instead of
|
||||
adjacent noise.
|
||||
|
||||
**Source those hints from the framework knowledge the project ALREADY ships** — the semantic facts taxonomy
|
||||
(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework WITHOUT
|
||||
Use the project's existing framework knowledge for those hints. The semantic facts taxonomy
|
||||
(`ruleset/semantic/*.toml`, `ruleset/frameworks.toml`) encodes sources/sinks/sanitizers per framework without
|
||||
user annotations ([[php-detection-is-engine-only]] zero-annotation value). Where framework knowledge covers the
|
||||
API, feed it as a hint; where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary
|
||||
and, on repeated confirmation, it graduates into the facts (the #2 feedback loop). No new annotation burden on the
|
||||
user. This is now a first-class part of the "#2 guided by #1" workstream, not a deferred nicety. (DeepSeek
|
||||
API, feed it as a hint. Where it does not (library APIs like banks/bedrock), the LLM proposes the trust boundary,
|
||||
which becomes a fact after repeated confirmation (the #2 feedback loop). No new annotation burden on the
|
||||
user. This is now part of the "#2 guided by #1" workstream and must not be deferred. (DeepSeek
|
||||
`deepseek-flash` is the working default; GLM 5.3 "cyber" model is future-spec.)
|
||||
|
||||
## POOL-P1 LABEL AUDIT (LLM, 2026-10-12) — 47% of labels are sound
|
||||
|
||||
Widened the banks label-mislocation check across all **49 public** pool vulns via DeepSeek (advisory + head code +
|
||||
The banks label-location check was extended to all **49 public** pool vulns via DeepSeek (advisory + head code +
|
||||
pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-12-pool-p1-label-audit/`.
|
||||
- **label_sound = 23/49 (47%)**; 53% unsound (25 pre-exist in base, 16 sink outside changed lines). Cost $0.038.
|
||||
- By family (sound/total): injection 10/13, access_control 5/8, untrusted_destination 4/7, deserialization 1/2,
|
||||
config_secrets 1/4, **path 1/7, output_encoding 1/8**. Injection labels mostly sound; path/output_encoding mostly
|
||||
mislocate or pre-exist.
|
||||
- **record 07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook** — on the ~53%
|
||||
- Record 07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook. On the ~53%
|
||||
pre-existing/mislocated vulns a change-scoped hook correctly stays quiet. The honest catchable denominator is the
|
||||
sound subset; record 07 needs this caveat + a sound-subset recomputation.
|
||||
- **Fair pilot re-score:** only bedrock + graphiti (injection) of the 5 are sound; the engine missed both — the
|
||||
real detection gaps. The 23 sound targets are the fair local eval set for "#2 guided by #1".
|
||||
- Only bedrock + graphiti (injection) of the 5 pilot cases have sound labels. The engine missed both, so these
|
||||
are detection gaps. The 23 sound targets are the fair local eval set for "#2 guided by #1".
|
||||
- Next: run the agentic search-with-proof hunter on the sound targets (start bedrock + graphiti), size its
|
||||
per-vuln call count/cost, measure catch + false-alarm. Then recompute record 07 on the sound subset.
|
||||
|
||||
@@ -384,13 +383,13 @@ pool label → sound/unsound). Full artifact: `benchmarks/measurements/2026-10-1
|
||||
Ran `ousast scan --mode deep` (DeepSeek `deepseek-flash`, model layer + tool_hunter, 60-call/8-hotspot bound) on
|
||||
bedrock + graphiti, the two sound pilot targets the static engine missed. Artifact:
|
||||
`benchmarks/measurements/2026-10-12-hunter-sound-targets/`.
|
||||
- **hunter 1/2 vs static engine 0/2.** graphiti: **HIT** — `falkordb_driver.py:207`, tag `family:cypher-injection`,
|
||||
- **hunter 1/2 vs static engine 0/2.** graphiti: **HIT**, `falkordb_driver.py:207`, tag `family:cypher-injection`,
|
||||
rationale traces `build_fulltext_query` concatenating user input into a Cypher string at line 278 (inside the
|
||||
labelled function, location-matches). bedrock: **miss** — 12 suspicions on adjacent concerns, never named the
|
||||
labelled function, location-matches). bedrock: **miss**, 12 suspicions on adjacent concerns, never named the
|
||||
`install_packages`→`pip` command injection.
|
||||
- **Cost is a non-issue:** ~$0.007/repo, 64 model calls for a whole-repo deep scan.
|
||||
- **Everything stopped at `suspicion`** — the proof arm did not elevate to corroborated/entailed/execution_confirmed.
|
||||
Release-grade BLOCK needs *proven* findings; measuring the proof arm's lift is the next depth.
|
||||
- Cost is low: ~$0.007/repo, 64 model calls for a whole-repo deep scan.
|
||||
- Everything stopped at `suspicion`. The proof arm did not elevate to corroborated/entailed/execution_confirmed.
|
||||
Release-grade BLOCK needs *proven* findings; the next step is to measure the proof arm's lift.
|
||||
- bedrock's miss is the concrete case for the DSPy prompt/hints loop above.
|
||||
|
||||
Next: (a) measure the proof arm (can it elevate graphiti's suspicion to proven?); (b) stand up the DSPy hint loop
|
||||
@@ -399,37 +398,37 @@ run for the release-relevant shape.
|
||||
|
||||
## STRATEGIC PIVOT: core problem = unprovable reachability; FIX THE EVALUATION TARGET FIRST (2026-10-12)
|
||||
|
||||
**Core problem (named with the maintainer):** the tool can SPOT dangerous ops but cannot prove they are
|
||||
ATTACKER-REACHABLE — the trust boundary lives outside the changed library code, in the caller/application a
|
||||
change-scoped scan does not see; and the project rightly won't emit unproven alarms. Recall → ~0 from unprovable
|
||||
reachability, not blindness. Cost/latency/engine-wall/corpus-labels were all downstream noise.
|
||||
The maintainer and team identified the problem: the tool can find dangerous operations but cannot prove
|
||||
an attacker can reach them. The trust boundary is outside the changed library code, in a caller/application
|
||||
that a change-scoped scan does not see. The project will not emit unproven alarms. Recall → ~0 from unprovable
|
||||
reachability, not blindness. Cost/latency/engine-wall/corpus-labels did not explain this problem.
|
||||
|
||||
**Chosen bet (of three — fix-target / advisory-with-conditions / build-prover): FIX THE EVALUATION TARGET FIRST.**
|
||||
A pre-push hook runs inside a developer's APPLICATION (entry points present, request→sink in-repo), but the pool is
|
||||
The chosen bet is to fix the evaluation target first. The three options were fix-target / advisory-with-conditions / build-prover.
|
||||
A pre-push hook runs inside a developer's application (entry points present, request→sink in-repo), but the pool is
|
||||
library-heavy, so we measured the hardest/wrong case.
|
||||
|
||||
**Shape classification (LLM, 49 public vulns, $0.033 — `benchmarks/measurements/2026-10-12-pool-p1-shape-classification/`):**
|
||||
reachability_in_repo True only 12/49 (24%); 37/49 are pure libraries. A legitimate catch needs sound label AND
|
||||
reachability in-repo → **5/49 (~10%)**: graphiti, desktopcommandermcp, malla, document-merge-service, mcp-remote.
|
||||
**Shape predicted the hunter result** (graphiti app-shaped→caught; bedrock library→missed). So record-07's 3/92 is
|
||||
scored against a ~90%-illegitimate denominator — the near-zero catch is largely a TARGET MISMATCH.
|
||||
Shape classification (LLM, 49 public vulns, $0.033, `benchmarks/measurements/2026-10-12-pool-p1-shape-classification/`):
|
||||
reachability_in_repo True only 12/49 (24%); 37/49 are pure libraries. A valid catch needs a sound label and
|
||||
in-repo reachability → 5/49 (~10%): graphiti, desktopcommandermcp, malla, document-merge-service, mcp-remote.
|
||||
Repository shape predicted the hunter result (graphiti app-shaped→caught; bedrock library→missed). So record-07's 3/92 is
|
||||
scored against a ~90%-illegitimate denominator. The near-zero catch largely comes from a mismatch between the targets and the tool.
|
||||
|
||||
**Next:** deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch rate on the corrected set; then
|
||||
Next: deep scan (regex/engine/hunter) on the 5 fair targets → per-tier catch rate on the corrected set; then
|
||||
stand up an app-shaped eval set from known-vulnerable apps (VAmPI/NodeGoat/DjanGoat/vibe-py) and recompute the
|
||||
floor. Deferred bets (advisory-with-conditions, reachability prover) revisited once the fair-target numbers land.
|
||||
floor. Revisit the deferred bets (advisory-with-conditions, reachability prover) once the fair-target results are available.
|
||||
|
||||
## FAIR-TARGET RESULT: detection 3/5 on app-shaped+sound (vs ~0 library-heavy) — target-fix validated (2026-10-12)
|
||||
|
||||
Deep scan on the 5 app-shaped+sound targets (`benchmarks/measurements/2026-10-12-hunter-fair-targets/`):
|
||||
- **3/5 detected**: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
|
||||
overlay — "outbound.request reached from parameter" on readFileFromUrl; the param-as-source mode fired because an
|
||||
entry point is present). **2/5 missed**: malla (DOM/template XSS) and dms (Jinja SSTI) — both in families
|
||||
families.toml DOCUMENTS as limited (template-mediated escaping).
|
||||
- vs the library-heavy pilot's ~0 → **the target mismatch was real and material.** On an app-shaped repo the engine
|
||||
- 3/5 detected: graphiti (hunter, Cypher injection), mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE
|
||||
overlay, "outbound.request reached from parameter" on readFileFromUrl; the param-as-source mode fired because an
|
||||
entry point is present). 2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families
|
||||
documented as limited in families.toml (template-mediated escaping).
|
||||
- The library-heavy pilot found ~0 → this confirms that the target mismatch affected detection. On an app-shaped repo the engine
|
||||
itself detects (parameter is a recognized source once the entry point is in-repo).
|
||||
- **But everything stops at `suspicion`/`static_corroboration`** — none proven. Cost ~$0.007/repo.
|
||||
- Everything stops at `suspicion`/`static_corroboration`. None are proven. Cost ~$0.007/repo.
|
||||
|
||||
**Strategy verdict:** bet #1 (fix the eval target) is validated as NECESSARY and MATERIAL but NOT SUFFICIENT.
|
||||
Bet #1 (fix the eval target) is necessary and improves detection, but is insufficient on its own.
|
||||
Detection appears on fair targets; release-grade BLOCK still needs (a) proof-arm elevation (suspicion→proven) and
|
||||
(b) family coverage (template XSS/SSTI) + the DSPy hint loop for misses. Next: recompute a floor on an app-shaped
|
||||
sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure the proof arm's suspicion→proven lift.
|
||||
@@ -438,53 +437,53 @@ sound subset / real app corpus (VAmPI, NodeGoat, DjanGoat, vibe-py), and measure
|
||||
|
||||
Measured whether the proof arm can elevate the 3 fair-target suspicions (graphiti, mcp-remote, dcmcp) to
|
||||
`exploit_demonstrated`.
|
||||
- **Correction:** the search arm's `FAMILY_ORACLES` (search/demo.py) differs from families.toml's `verifier`.
|
||||
All 3 hits HAVE a runtime oracle: injection→(sql,command), untrusted_destination→(ssrf). (deserialization/
|
||||
access_control/config_secrets have NONE — empty tuple → `no_oracle`.) So dcmcp IS provable in principle
|
||||
- The search arm's `FAMILY_ORACLES` (search/demo.py) differs from families.toml's `verifier`.
|
||||
All 3 hits have a runtime oracle: injection→(sql,command), untrusted_destination→(ssrf). (deserialization/
|
||||
access_control/config_secrets have none, empty tuple → `no_oracle`.) So dcmcp is provable in principle
|
||||
(earlier "unprovable" was a families.toml misread).
|
||||
- **But proving is a runtime differential:** `search/verify.verify()` requires (1) an AUTHORED declarative demo
|
||||
- Runtime differential proof through `search/verify.verify()` requires (1) an authored declarative demo
|
||||
(start the app on a port, run exploit steps, fill oracle placeholders: sql→database_path, command→marker_dir,
|
||||
ssrf→callback_url), (2) BOTH head AND base runnable with OFFLINE deps (`pip --no-index`, `npm ci --offline`),
|
||||
ssrf→callback_url), (2) both head and base runnable with offline deps (`pip --no-index`, `npm ci --offline`),
|
||||
(3) a sandbox differential "owned effect in 3/3 affected runs and 0/3 safe runs". Local host has a sandbox
|
||||
(`isolation_mode: userns`); the plane supplies a task-boundary dispatcher.
|
||||
- **The scan/push path reaches only `suspicion`/`static_corroboration`** (verification.py levels) — there is NO code
|
||||
- The scan/push path reaches only `suspicion`/`static_corroboration` (verification.py levels). There is no code
|
||||
path from a scan suspicion to `exploit_demonstrated`, and no caller wires the search Coordinator from the push
|
||||
flow. So **proof-arm lift at push time = 0/3**: not because the vulns are unreal, but because proving is a
|
||||
per-target, offline-build, runtime, differential operation that cannot run inside a pre-push hook. (Prior
|
||||
per-pair demo attempts were mostly inconclusive on this exact wiring — app startup, offline deps, oracle plumbing.)
|
||||
flow. So proof-arm lift at push time = 0/3: not because the vulns are unreal, but because proving is a
|
||||
runtime differential operation with an offline build for each target that cannot run inside a pre-push hook. (Prior
|
||||
per-pair demo attempts were mostly inconclusive on this exact wiring, app startup, offline deps, oracle plumbing.)
|
||||
|
||||
**Strategic consequence (resolves the three-bet fork):** proof/BLOCK is an ASYNC capability (CI / plane / triage
|
||||
with authored demos + a runnable sandbox), NOT a pre-push one. At push time the honest ceiling is corroborated
|
||||
suspicion. So:
|
||||
- **Bet #2 (advisory-with-conditions) is the honest PRE-PUSH product**: surface corroborated suspicions ("change
|
||||
This resolves the three-bet choice. Proof/BLOCK runs asynchronously (CI / plane / triage
|
||||
with authored demos + a runnable sandbox). At push time, the strongest available evidence is corroborated
|
||||
suspicion:
|
||||
- Bet #2 (advisory-with-conditions) is the honest PRE-PUSH product: surface corroborated suspicions ("change
|
||||
introduces a flow that looks like X; reachable if untrusted Y reaches it"), let the dev / coding-agent (who has
|
||||
the app context) judge — matching the "oversee coding agents" framing and the suspicion-rung reality.
|
||||
- **Bet #3 (the proof arm) is the ASYNC BLOCK lane**: run `search/verify` on the plane/CI for the suspicions worth
|
||||
proving, with authored (eventually DSPy-authored) demos. Worth an empirical run on one tractable target
|
||||
(mcp-remote) to confirm the arm demonstrates on a real unseen vuln — a bounded, separate effort.
|
||||
the app context) judge. This matches the "oversee coding agents" purpose and the evidence available at the suspicion rung.
|
||||
- Bet #3 (the proof arm) is the ASYNC BLOCK lane: run `search/verify` on the plane/CI for the suspicions worth
|
||||
proving, with authored (eventually DSPy-authored) demos. Run a bounded, separate test on one tractable target
|
||||
(mcp-remote) to confirm that the arm demonstrates a real unseen vuln.
|
||||
|
||||
## BANKS FIXED-SEARCH MVP (2026-10-12) — "no wrong targets, only wrong searching" demonstrated
|
||||
|
||||
Record: `benchmarks/measurements/2026-10-12-banks-fixed-search-mvp/`. Target: banks (the "library" that gave
|
||||
the engine ZERO path findings in the pilot). Same target, same CPG — only the search changes.
|
||||
no engine path findings in the pilot). The target and CPG are unchanged; only the search changes.
|
||||
|
||||
- Injected the two annotations a per-target LLM steer would supply: **parameter-as-source**
|
||||
(`parameter_sources=True`, so `prompt.name` counts as untrusted) + the **missing file-write sink**
|
||||
(`write_text`/`read_text` added to path sinks).
|
||||
- **Baseline (shipped search): 0 resolved `prompt → write_text` flows** — its 5 flow rows are all census/
|
||||
- The shipped search baseline resolved 0 `prompt → write_text` flows. Its 5 flow rows are all census/
|
||||
evidence (`sink=None`), matching the pilot's zero path findings.
|
||||
- **Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
|
||||
`from_prompt_path`** (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear ONLY with
|
||||
- Fixed search: the engine resolves `prompt → path-join → prompt_file.write_text(prompt.raw)` in
|
||||
`from_prompt_path` (`source=prompt`), plus a second `write_text` sink in `_scan`. These appear only with
|
||||
the annotations.
|
||||
- **Thesis demonstrated on a concrete case:** banks was not a wrong target; the search was unannotated. The
|
||||
fix needed exactly **two** inputs — the param-source and the file-write sink — which are precisely what the
|
||||
- The banks result supports the hypothesis: the target was valid, but the search lacked annotations. The
|
||||
fix needed exactly **two** inputs: the param-source and the file-write sink. These are what the
|
||||
per-target LLM steer (DSPy-optimised, framework-seeded) has to learn to supply. This is the first measured
|
||||
instance of the "#2 guided by #1" loop closing on an unseen target.
|
||||
- Caveats: (1) one row shows `source=path` not `prompt` — the param-source fires broadly, a precision knob for
|
||||
the steering loop; (2) this is a TRACE, not the runtime differential proof (proof stays the async BLOCK lane).
|
||||
instance of a complete "#2 guided by #1" loop on an unseen target.
|
||||
- Caveats: (1) one row shows `source=path` not `prompt`, the param-source fires broadly, the steering loop needs to control that breadth to improve
|
||||
precision; (2) this is a trace, not the runtime differential proof (proof stays the async BLOCK lane).
|
||||
|
||||
**Instrument note / filed bug:** the first MVP used the single-query `cpg.run("taint", …)` path and returned
|
||||
0/0 — an INSTRUMENT FAILURE, not a result (`cpg query taint failed` on every query). Root cause filed in
|
||||
The first MVP used the single-query `cpg.run("taint", …)` path and returned
|
||||
0/0 because of an instrument failure, rather than producing a valid result (`cpg query taint failed` on every query). Root cause filed in
|
||||
`bugs/taint-sc-single-query-signature-skew.md`: `request_params()` emits 5 keys (`guards`, `fixedOrigin`,
|
||||
`originAnchors`, `quotedSanitizers`, `fieldParameterSources`) that `taint.sc`'s `exec` signature does not
|
||||
declare, so both single-query sub-routes (CLI `--param` and session named-arg) reject them. The batch path
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Valid detection benchmark
|
||||
|
||||
Created 2026-10-12. This brief **supersedes the frozen-SZZ-pool evaluation** (`unseen-repo-evaluation`), which is retired.
|
||||
SZZ guessed its labels and placed the vulnerability in the wrong location about half the time (audit: path 1/7,
|
||||
output-encoding 1/8 sound). Also, ~76% of entries are library-internal (reachability not in-repo). Catch /
|
||||
false-alarm numbers from that pool are not valid benchmark results. This brief defines a benchmark with
|
||||
sound ground truth and metrics that engineers and managers can understand.
|
||||
|
||||
## The two questions the benchmark answers
|
||||
|
||||
A pre-push security check must answer two questions:
|
||||
|
||||
1. **If I'm about to push a real vulnerability, does it stop me?**
|
||||
2. **If my change is fine, does it leave me alone?**
|
||||
|
||||
The benchmark runs the **whole tool** as a developer uses it at push time. Quick rules and the engine are on,
|
||||
as is the LLM-guided arm. The repositories are ones the tool has **never seen**.
|
||||
|
||||
## What makes the data valid
|
||||
|
||||
### Positive cases — "should catch" (real, verified, reachable vulnerabilities)
|
||||
- Take the label from the security fix, never from SZZ. Use the code *just before* the fix, at the file/function
|
||||
the fix changed. The fix establishes the vulnerability location without a blame walk-back or mislocation.
|
||||
- Keep only app-shaped cases where the vulnerability is reachable from an entry point *inside the repo*. This
|
||||
can be a web route or a request handler; a CLI argument also qualifies. Drop library-internal vulnerabilities
|
||||
that depend on a caller outside the repo. A pre-push tool cannot see that trust boundary, so scoring it is unfair.
|
||||
- Verify each case through LLM adjudication and a human spot-check. Confirm (a) that the pre-fix code is
|
||||
vulnerable and (b) that the fix addresses it. Also confirm (c) that it is reachable in-repo.
|
||||
|
||||
### Negative cases — "should not alarm" (ordinary safe changes)
|
||||
- Use everyday commits from the **same** application repositories that are not security fixes. They are plentiful
|
||||
and cheap to collect. Almost every real push is ordinary, so false alarms determine whether developers keep
|
||||
using the tool. The negative set must be much larger than the positive set.
|
||||
|
||||
### Held out
|
||||
- None of these repositories were used to build, train, tune, mine or harvest the tool.
|
||||
|
||||
A few dozen vulnerabilities with sound labels and in-repo reachability, each verified, are better than hundreds
|
||||
of SZZ guesses. SZZ is wrong about half the time. Curating this set takes manual work, supported by the LLM-audit
|
||||
tooling that found the SZZ pool's defects. Dozens of good cases are enough.
|
||||
|
||||
## Metrics someone understands
|
||||
|
||||
| Metric | Plain-English question | Example |
|
||||
|---|---|---|
|
||||
| **Catch rate** (recall) | Of real vulnerabilities, how many did it flag? | "Caught 18 of 20 (90%)" |
|
||||
| **False-alarm rate** | On safe changes, how often does it cry wolf? | "Flagged 3 of 100 ordinary pushes" |
|
||||
| **Precision** | When it alarms, how often is it real? | "When it speaks up, it's right 8 of 10 times" |
|
||||
| **Proof rate** | Of what it flags, how much can it *demonstrate*, not just suspect? | "12 of 18 catches came with a working demonstration" |
|
||||
|
||||
Report false alarms as **"per 100 pushes"**, which reflects a developer's experience. Vulnerabilities are rare
|
||||
among real pushes. Precision depends on that ratio, so a 50/50 test set overstates it.
|
||||
|
||||
Report the first two metrics together:
|
||||
|
||||
> "On unseen application repos: catches 90% of real vulnerabilities while false-alarming on 3% of ordinary
|
||||
> pushes."
|
||||
|
||||
This wording is easier for a manager to understand than "3/92, Wilson95 1.1–9.2%".
|
||||
|
||||
## The gold-standard metric — and the project's edge
|
||||
|
||||
**Proved catches** do not depend on labels. The search-with-proof / differential-exploit arm demonstrates the
|
||||
bad effect on the vulnerable code and its absence on the fixed code. The demonstration *is* the ground truth.
|
||||
The headline to aim for is:
|
||||
|
||||
> "On unseen app repos: caught and *proved* N of M real vulnerabilities, with K false alarms per 100 ordinary
|
||||
> pushes."
|
||||
|
||||
The architecture is built to produce this kind of benchmark.
|
||||
|
||||
## Why the old approach was not feasible (recorded so it is not re-tried)
|
||||
|
||||
- SZZ ground truth is machine-guessed and ~half wrong → every catch/FP number off it is noise.
|
||||
- Library-heavy corpus → most cases are unreachable in-repo, so a pre-push tool is scored on things it
|
||||
structurally cannot see.
|
||||
- Scoring regex-only (engine and LLM off) measured a tier nobody would ship alone.
|
||||
- Automated harvesting prioritized pool size over sound ground truth and in-repo reachability.
|
||||
|
||||
## Open questions for the spec phase
|
||||
|
||||
- How many verified positives are enough for a credible catch-rate interval (target set size)?
|
||||
- The negative:positive ratio that reflects real push traffic (for an honest false-alarms-per-100-pushes).
|
||||
- Curation workflow: how LLM adjudication + human spot-check are recorded and frozen per case.
|
||||
- Language/framework coverage of the positive set (be explicit about what is and isn't represented).
|
||||
Reference in New Issue
Block a user