Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.
Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 hits have a runtime oracle (search FAMILY_ORACLES: injection->sql/command, ssrf for
untrusted_destination -- corrects the earlier families.toml-based 'dcmcp unprovable'). But
search/verify.verify() needs an AUTHORED runnable demo + both head/base building OFFLINE + a
sandbox differential (3/3 affected vs 0/3 safe). The scan/push path reaches only suspicion/
static_corroboration; no code path from a scan suspicion to exploit_demonstrated, no Coordinator
caller from the push flow. So proof-arm lift AT PUSH TIME = 0/3 -- proving is a per-target,
offline-build, runtime, differential operation that can't run inside a pre-push hook (prior
per-pair demos were mostly inconclusive on this wiring).
Strategic consequence: proof/BLOCK is an ASYNC lane (CI/plane/triage with authored demos), not
pre-push. Honest pre-push product = advisory-with-conditions (bet #2); proof arm = async BLOCK
lane (bet #3). Local sandbox exists (userns).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran ousast scan --mode deep on the 5 app-shaped + sound pool targets (the fair change-scoped
eval set from the shape classification). Detection 3/5: graphiti (hunter, Cypher injection),
mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE overlay -- 'outbound.request reached
from parameter' on readFileFromUrl; parameter-as-source fired because an entry point is present).
2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families families.toml
documents as limited.
This validates the strategic pivot: the near-zero catch was substantially a TARGET MISMATCH --
on app-shaped repos the engine itself detects (0 -> 3/5). But all hits stop at suspicion/
static_corroboration; release-grade BLOCK still needs proof-arm elevation + family coverage +
the DSPy hint loop. Cost ~$0.007/repo. So bet #1 (fix eval target) is necessary and material
but not sufficient.
Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat,
DjanGoat, vibe-py); measure proof-arm suspicion->proven lift.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Named the core problem with the maintainer: the tool spots dangerous ops but cannot prove
attacker-reachability -- the trust boundary lives outside the changed library code. Chosen
strategic bet (of three): FIX THE EVALUATION TARGET FIRST.
LLM shape classification of the 49 public pool vulns ($0.033): reachability_in_repo only
24% (12/49); 76% are pure libraries. A legitimate change-scoped catch needs sound label AND
reachability in-repo -> just 5/49 (~10%): graphiti, desktopcommandermcp, malla,
document-merge-service, mcp-remote. Shape predicted the hunter outcome (graphiti app-shaped ->
caught; bedrock library -> missed). So record-07's 3/92 is scored against a ~90%-illegitimate
denominator -- the near-zero catch is largely a target mismatch.
Next: deep scan on the 5 fair targets for a per-tier catch rate; then an app-shaped eval set
from known-vulnerable apps. Artifacts: 2026-10-12-pool-p1-shape-classification/ (+ earlier
label-audit, hunter-sound-targets). Memory: evaluation-target-mismatch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed:
- graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces
build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the
labelled function, location-matches). The static engine produced zero findings here.
- bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never
named the install_packages -> pip command injection.
Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything
stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs
proven findings.
Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO,
pluggable; evolving field) whose objective is to learn what hints/annotations steer the model
to the real sink, sourced from the framework knowledge already shipped (no new user
annotation), with LLM-proposed boundaries graduating into the facts.
Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen
the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek
(advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%)
sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped
hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7,
output_encoding 1/8, config_secrets 1/4.
Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta
hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES).
Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine
missed both = the real detection gaps for the LLM-guided hunter.
Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek
is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
DeepSeek deepseek-flash reachable: ~$0.0002/call, ~1.1s (sub-cent/push). The LLM gives the
correct verdict WHEN aimed at the real sink + source context: asked about the pool's
SZZ-labelled site (_load) it correctly says not-vulnerable; asked about the real sink
(from_prompt_path line 44, prompt.name request-derived) it correctly says vulnerable. So '2
guided by 1' works as a mechanism.
Two blockers, both concrete: (1) it needs the agentic search-with-proof hunter to supply
caller/source context -- single-shot on the ranker's region is insufficient; (2) the pool's
banks label is mislocated (head_lines 30/104/138-141, function _load; real sink is line 44),
so even a correct finding wouldn't score as a catch and the sink pre-exists in base -- a
change-scoped hook correctly stays quiet. Part of the pilot engine 0/5 is SZZ mislocation, not
a detection failure. GLM/abliteration fallback errors HTTP422 on the thinking param (client
fix needed; DeepSeek works).
Next: validate the 5 pilot labels (change-located & introduced?) before detection runs; then
size the agentic hunter on the sound targets; feed confirmed source shapes back as guarded facts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified root cause: the pilot repos are libraries whose untrusted input is public-API
parameters, not HTTP requests. Python taint sources (ruleset/semantic/python.toml) are web/CLI
only; the parameter-as-source mode (_parameters_are_input) fires only for recognised entry
points, and the pilot vuln regions are entry:false -- so their params are deliberately not
sources. Widening that statically would reintroduce the WordPress/PMPro false-SQL-injection
flood the docstring documents. So the static engine cannot safely source library APIs on its
own -- which is exactly the seam for the maintainer's '2 guided by 1' directive: the LLM
adjudicates which library parameter is a trust boundary, the taint engine traces that
source->sink as proof (not LLM say-so), and repeated source shapes graduate to guarded static
facts. First increment (pending budget): LLM-guided-source mode on the 5 pilot regions,
measuring source naming, traced finding, catch lift vs regex 1/5, and false-alarm on ordinary
changes; staged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inspecting head.findings (not ranker evidence) overturns the previous commit's read:
the dataflow/taint tier produced ZERO findings in the vuln families (path/injection/deser)
on all 5 repos -- only config_secrets (banks 1, socketio 2; launch-editor/graphiti/bedrock 0).
The tier1/score2.0 'sink' I cited was the RANKER's static candidate, not a confirmed taint
finding. taint_evidence_not_applicable = no source->sink flow was traced (banks: sink found,
no source). So ambiguous_line_correspondence is downstream and moot -- there are no
vuln-family findings to anchor, and the region-level witness fix (Option A) is refuted before
implementation. The LLM adjudication tier was disabled (model_calls=0).
Reprioritised: #1 LLM-tier pilot L1/L2 (now primary, staged, costs money), #2 taint
source/sink coverage + region ranking, #3 change-anchoring (only after tiers yield findings).
Instrument rule: caught the ranker-vs-finding misread before building the wrong fix.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Five public pool repos, vuln change each, clean host, 300s. The shipped engine added NO
catches over the regex floor. 3/5 repos finished their region queue (deferred=0) and still
emitted no finding, so change-scoping is not the path to catch. The engine ranked the vuln
region in all 5 and found the right sink on 2 (score 2.0), but withholds any finding it
cannot anchor 1:1 to changed lines -- all 5 hit ambiguous_line_correspondence ->
witness_unresolved -> coverage incomplete, suppressing even the scored sinks. Confirmed not a
shallow-clone artifact (correspondence is computed from both blobs).
Reprioritised engine levers: #1 change-anchoring robustness (region-level witness fallback),
#2 family ranking/taint coverage, #3 the LLM adjudication tier (the pilot is the case for it).
Release read: the pre-push net is NOT releasable on detection yet.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On a clean host the shipped engine runs dataflow (query 0->35s, selected 0->356 regions).
The evidence/taint batch (~225s) is invariant to warm session and graph preload; importCpg
is cheap (~35s). The real barrier is taint.sc's ~160s per-batch flow work, so build+evidence
~255s of a 300s budget defers 1009/1365 regions. Lever = change-scope the family/file set the
taint batch walks, not session/preload. Corrects the prior ~200s graph-load wall diagnosis.
Next decisive experiment: engine-on unseen pilot vs record-07 regex floor (catch 3/92).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified on the records: the push replay arms hardcode model="disabled",
max_model_calls=0, cost_usd=0 (NO LLM) and engine_covered=0 (engine never ran,
~200s wall). So the measured catch~2.5% is the quick pattern-rule tier ALONE --
grep-with-structure. Pattern rules alone will never be enough.
Reprioritize G2 after record 07 to an efficiency barrier analysis over the two
tiers that carry detection and both sat out:
- engine (Joern dataflow): E1 split the ~150s evidence-fixed (JVM/graph-load/
census); E2 does a warm session + graph preload at build kill it (decisive);
E3 analysis per-pair; E4 build vs repo size.
- LLM adjudication: L1 model latency + calls per push; L2 recall lift over
regex+dataflow on the pool.
Deliverable: a barrier table (budget x tier combo -> fits? measured recall/FP)
for a realistic pre-push window (<=120s target, 300s hard) -> minimal releasable
config. Change-scoping branch is a late add-on (trims only the per-pair tail),
not the lever. Also fixed a stray </content> tag in the brief.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Records the full G2 engine-detection finding on main (code attempt parked on
branch engine-change-scoped-evidence-pass, unmerged). The engine arm has ~200s
intrinsic fixed overhead -- ~48s CPG build + ~150s first-query graph-load+census
(measured uncontended: build 48s, evidence 243.79s over 1015 pairs, query 0) --
so a 300s budget cannot reach the flow queries. Pair-count change-scoping trims
only the ~86s per-pair tail; the warm-session env times out at startup contended
and spares only subsequent batches. Real fix is clean-host engine work (reliable
session startup + graph preload, or census-cost reduction). Parked to finish G0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Saved-artifact run: total=291s build=52s query=0s arbitrate=238.75s, and
query_seconds_by_kind[evidence]=238.54s. The evidence-tier pass is the whole
middle; affected_context + _scope_work are 0.21s. It tiered 1015 pairs at
~235ms/pair (contended) and exhausted the budget -> selected=0, deferred=1365
all deadline_exhausted. Secondary anomaly: ~50-100x the design's assumed
2.5ms/pair -- the evidence query is pathologically slow on a cold large CPG.
Increment 2's single target is the evidence pass (scan.py:338-359). Fork
recorded: (i) scope+bound the pass change-first, (ii) two-phase neighbourhood
pass, (iii) per-pair speedup. Must preserve reach-into-unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A first scoping attempt (restrict analyzed regions to changed files) was reverted:
it breaks a tested design invariant. The engine deliberately selects an unchanged
region's question when a changed region reaches it
(test_changed_dependency_reaches_unchanged_ranked_question;
test_change_does_not_replace_ranker_selection). _changed_files feeds _evidence_pass
only as contextFilter/contextPaths (itemise vs summarise), not to shrink the tier
population. Any scope must keep changed regions PLUS the regions they reach.
Also: increment 1 over-claimed. Increment 0 measured build=67s/query=0s but NOT
the split of the ~236s middle (evidence-tier pass vs affected_context/context
resolution vs _scope_work enumeration). Increment 2 must capture per_kind /
other_seconds / affected_context timing and cut the dominant phase while
preserving reach-into-unchanged -- instrument before fixing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 4095 candidates are not an in-scan artifact: push/runner.py:345 discovers
head_regions across the whole head tree, then runner.py:397 passes them wholesale
to scan_repository with change_context threaded in but unused for scoping. The
scan enumerates all regions x families (scan.py:167/364) and, with
order_by_evidence=True, evidence-tiers every pair before analysis -> the ~236s
from increment 0 -> deadline at query_seconds=0.
Fix locus (increment 2): scope the analyzed region set to context.changed_paths +
dataflow neighbourhood (regions.py:279 primitive exists; compare_targeted_base is
precedent). Keep the CPG whole-repo so taint can trace into the change from
unchanged code; only the question set is change-scoped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Timed live bedrock replay: build_seconds=66.69, query_seconds=0.0, total 303s,
4095 candidate questions all analysis_not_started=deadline_exhausted, selected=[],
regions_scanned=0. ~67s builds the CPG; ~236s enumerates/ranks/prepares 4095
repo-wide candidates for a 3-file change, then the deadline hits before any query
runs. The CPG build is not the bottleneck; the over-broad candidate set is ->
change-scoping is the confirmed lever (and compounds with the correspondence fix,
since per-candidate prep includes correspondence work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Track the engine-detection lever found while driving G0. Live diagnosis on the
frozen pool: the engine's "no finding" on unseen changes is usually
analysis-never-runs -- it ranks repo-wide candidates and defers them all
(deadline_exhausted) before analyzing the pushed change (bedrock injection,
303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis
from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so
it runs within budget.
- .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured),
hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis
cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness
+ tests -> 4 live re-test -> 5 measure delta on the engine arm).
- roadmap.md G2: active workstreams (change-scoped analysis; the done-but-
necessary-not-sufficient line-correspondence fix), for progress tracking.
Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule
record-07. Coverage also needs the capability registry (Task 8.3); arm-scale
measurement needs the kube lane.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
