The full suite after the replay found regressions from 062a7b7 and 94e96c6:
- catalog_gen and candidates.labeled_spans still unpacked 3-tuples and expected None on a parser
rejection; source.ranges returns FunctionRange (4 fields, module entry first) and raises ValueError.
- ranges_from_ir passed the language to declarator_name, so the repo-facts declaration table named a
callback call (exec(..., function () {) as a function; the pre-P1 call had no language.
- the catalog_gen fail-loud test blocked the old module name.
- docs: a renumbered examples anchor, and two .kiro paths in the public threat model.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
DSPy 3.4.1 raises AdapterParseError for an unparseable ReAct step, which is not the ValueError its loop
catches, so one malformed step discarded the whole hunt (7 of 60 hunts in the repeats). The step now ends
the loop like an invalid tool choice. Records carry loop_parse_failed beside parse_failed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Committed before the repeats finish. Unit is (change, repeat); paired bootstrap
over changes and exact McNemar from openultrasast.stats; failures stay in the
denominator. Dry run on the existing single runs reproduces 6/12 and 3/12.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
3 of 12 with the fixed answer check against the pilot hunter's 6 of 12, one
run each, not significant. The intended zero-cost replay was not a replay:
answers at temperature 0 differ between runs and the cache did not serve the
second run. Programs must be compared over repeated runs, paired by change.
About $1.10 for both runs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The answer check counted only the rendered function text as shown, so 7 of
8 baseline suspicions citing lines the model had read with read_file were
marked unknown (baseline 1/12 against the pilot hunter's 6/12). Tools now
record the lines they return. A DSPy parse failure scores as zero findings
with a flag instead of a broken run. The per-run response cache now takes
effect through dspy.configure_cache.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 of 12 positives; 12.6 false alarms per 100 ordinary pushes (2.1 on changed
lines only). The misses come from what the rules and facts express (access
control lives in obligations, which do not convert; sources cover request
objects only), not from the engine. Comparison arm only: a Semgrep miss is
never evidence against a hunter finding.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A Semgrep, quick-rule or engine match may enter the feature table as
positive evidence; its absence is never a feature, filter, demotion or
gate on a hunter suspicion, and no acceptance rule counts it. The Semgrep
arm is a comparison only and may at most replace the regex quick tier.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
openultrasast.stats: Wilson intervals (statsmodels), exact McNemar, a
repository-clustered percentile bootstrap and its paired form with one
quantile convention, grouped folds (GroupKFold, leave-one-group-out), and
per-column separation (ROC AUC) for the leak audit and the signal screen.
Checked against the benchmark scorer's Wilson and the learn harness's
McNemar. hunt/features.py turns the feature table into the engineered
frame (distances, counts, flags) and keeps the map from every engineered
feature to the annotation the model is shown, so an ablation mask drops
both, and screen() says which annotation carries signal before a model
call is spent. The [hunt] extra now carries the libraries; the core
install is unchanged. The learn harness's own copies stay until that line
is revived.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hunter-learning-loop groups 1-3 (first increment). The feature table is the
contract: one row per touched function with diff, source, semantic and
taxonomy columns, provenance per producer, labels by the benchmark's rule.
Producers read the repository at head only (a planted advisory id and
commit text never reach a feature). The hunt program is a DSPy ReAct loop
over a signature whose inputs are the unmasked columns in a fixed order;
answers outside the closed family list or citing unseen lines score as
unknown. The runner writes records the benchmark scorer reads, resumable
by change, program id and mask, and refuses the held-out pilot unless the
run is a report run. Research extra [hunt] = dspy, pandas; the core
install is unchanged. 43 tests on a scripted model, no network.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Engine LGPL-2.1, shelling out carries no obligation; the registry rules are
under the Semgrep Rules License since 2024-12-13 and cannot ship, only our
own converted rules can. Measured 4.7 s startup and 5 to 10 ms per file;
converted regex and taint rules fire for Python, PHP and JavaScript; C
propagation and Groovy do not; one sanitizer semantic we refuse on purpose
is accepted by Semgrep. Recommendation: a Semgrep arm on the valid pilot
first. Recorded in the hunter-learning-loop research note and breadth.toml.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
610 pair files, 570 labelled functions. lizard matches the parser engine on
C, Python and Java but loses JavaScript names and TypeScript end lines:
keep ours, take its cyclomatic complexity as a feature candidate.
tree-sitter-language-pack gives identical spans, is faster, replaces six
grammar wheels with one and adds Go, Ruby, PHP and Perl: adopt as grammar
packaging. Also found: .c files should try the c grammar before cpp.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Six groups: table and producers; the hunter as a program with the search
metric and the run artifact; runner integration and a baseline on the 12
pilot positives; training set, folds, paired ablation and the importance
table; proposals and the acceptance rule; the pilot gain report.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Feature engineering is the centre: a feature table with switchable columns
is the contract between producers (source.facts_for, semantic facts, entry
points, diff hunks; lizard and Semgrep only after their evaluations), the
ablation runner and the hunter program. DSPy runs the program (ReAct over a
HuntSignature; GEPA and MIPROv2 propose), our rule accepts on held-out
folds. The pilot stays held out; the hook, its client and the core install
are untouched. Requirements approved by the maintainer ('4 first').
Co-Authored-By: Claude Fable 5.1 <[email protected]>
src/openultrasast/source.py replaces the copies in semantic/functions.py,
plane/tasks/repo_facts.py, learn/features.py, learn/excerpt.py and
plane/tasks/verify.py. Two engines share one sentinel (<module>), one
DECLARATION/KEYWORDS/RESERVED set and one alias map: the parser engine
(ranges, enclosing, bounds, call_sites, callers_of, facts_for) for the
scorer, the overlay and the hunter's annotations; the line-pattern engine
(declared_functions, enclosing_at, function_span, function_bounds, the brace
matcher) that labels, features, roles, excerpts and plane facts were built
on. learn no longer imports plane.tasks. Frozen records that store the old
sentinel are mapped on read (module_function).
Gate: labels rebuilt before and after are identical, 7,921 rows, after
sentinel normalisation (benchmarks/measurements/2026-10-13-source-module-
label-diff/record.json). 490 targeted tests pass; 3,312 collect. Net: 657
lines deleted, 212 added. benchmarks/unseen keeps <global> for its own
frozen manifests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: the four hunter tools no caller enables (read_definition,
entry_points, flows, obligations) with their schemas and dispatch, the dead
mapping branch of find_refs; 20 functions whose only caller was one test
(mapping task records, compute_ranking_metrics, build_quick_hunter_prompt,
validate_evidence_transition, should_fuse, harness processors,
rank_targets_with_model, model/report slice rendering, taxonomy
hierarchical/related, config_value.verdict, taint.verdict, roles.load_roles,
features.language_of and build_for_scan, folds.leave_one_*,
features.recorded_case, search BatchExecutor and the three __main__ shims,
delivery.put_input); the receiver delivery mode of the plane. push_inputs.py
moved to benchmarks/push. memory.purge stays: the live-S3 test cleanup uses
it. Net: 2,730 lines deleted, 207 added. 473 targeted tests passed; 3,282
tests collect. breadth.toml records the removals.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep
now records a regress_retired degradation and equals standard; proof is the
search-with-proof arm's job), the sandbox docker runner and probe that only
regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers
only regress called; skills.py and index.py with the hunter pool's skill
routing; model/audit.py; cpg/queries/vocabulary.sc with
benchmarks/push/ontology_gap.py. Docs and the scan skill describe the
retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed,
28 skipped; 3,331 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/improve/ (1,175 lines), the ousast improve
subcommand, plane/tasks/loop.py and loop.yaml with the --loop generation,
the memory-snapshot step in memory.seed, pairs.build_pair_signals, and the
four improve tests plus the loop cases in other tests. Docs no longer
describe the lever. Net: 3,073 lines deleted, 83 added. Targeted tests:
154 passed; 3,436 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
PATHFINDER-2026-10-13/: feature inventory, ten per-unit flowcharts with
file:line evidence, cross-feature and within-unit duplication reports, a
proposal that keeps every package on one of the four active lines, and
copy-ready plan prompts in order. .kiro/breadth.toml: every spec and
capability with a status and the deletion that closes it. No code changed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
fossil-mcp dead-code over src found 72 candidates; 69 are referenced by tests,
__all__ or other modules, 3 are not and are removed (cli._first_source,
pairs._scorable, provider.openrouter._extract_message_content). Eight spec
directories that steering already called retired or superseded leave the tree
(history keeps them, commits named in overview.md): harnessx-removal,
harnessx-self-improving-rulesets, openrouter-sast-harness, learning-harness,
constrained-detector, three-stage-scan, real-world-vfc-slice,
unseen-repo-evaluation. overview.md gets a current-state section: the valid
benchmark numbers, what they say, and the four active lines in order.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The hunter gets an annotation model (named, versioned facts about the push)
and statistical ablation decides which annotations stay. Labels from fix
sites and ordinary pushes, training set apart from the held-out pilot,
compiled replayable program, train-and-holdout acceptance rule, two stages
reported separately, no thresholds set in this spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sweep over the pilot's hunter records: most 'vulnerable' suspicions on ordinary
pushes sit far from the diff. Scope to the change, then confidence, then the
config_secrets family; 5/12 recall kept at 8.8 false alarms per 100 (in-sample).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opt-in filter on the classifier verdict for hunter-sourced defects; the default
keeps everything. Each combined record notes the kept verdicts and the counts.
Live check on the pilot: (a)+(b) filtered to 'vulnerable' gives recall 5/12 and
54.4 false alarms per 100 ordinary pushes, matching the scratch numbers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Arm (b) hunter over all 252 changes: recall 6/12 raw, 5/12 with the classifier
filter; false alarms 92.5 and 54.4 per 100 ordinary pushes. Arm (a) push hook at
163/252: recall 0/12, 0.7 false alarms per 100. Per-positive table, classifier
stats, the one hook false alarm, and the disk fix for the replay output.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Live run: every classification failed with ValueError because Caller.ask
reads client.usage as a dict (the learn CLI always hands it a MeteredClient)
while a bare DeepSeekChatClient keeps usage as a list of rows. build_caller
now wraps a client that carries a usage list in MeteredClient; scripted and
already-metered clients pass through unchanged.
Regression test runs classify_family through the real DeepSeekChatClient
wrapper with a stub transport (the module fixture that stubs classify is
restored for it). The error field stays the exception type only, since a
message can carry a model key.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.
hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.
14 new offline tests, 67 green with the benchmark tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.
`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.
9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.
Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run of the valid detection benchmark. Labels come from the fix
site (fix^1), never SZZ.
- 32 held-out Python/JS advisories sourced from the public pool-p1 set,
CWE + text from OSV. 133 candidate locations -> 125 after dropping
tests/minified.
- 125 DeepSeek verification calls: 31 locations passed all three checks
(vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos.
- Maintainer spot-check accepted all 12.
- 20 ordinary negatives per positive change from the same repos, 240 total.
- Held-out guard: 0 dropped against the training-only used set.
- Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and
NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/.
Three fixes found by the live run:
- catalog.py: skip a site the fix renamed or removed instead of aborting.
- assemble.py: negative ratio counts positive changes, not accepted
functions (malla would have demanded 260 negatives). Test updated.
- held-out guard: run against training sources only; build_used() also
scans measurements/populations, which flagged the 12 repos via my own
SZZ-pool analysis outputs (circular).
Replay and metrics not yet run; see NOTES.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.
Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.
New offline modules under benchmarks/valid/:
- catalog.py positives labelled at the fix site (never introducing/last_touch)
- verify.py one LLM call per case (vulnerable? fix addresses it? reachable
in-repo?) + human spot-check; this is the app-shaped filter.
Honors the instrument rule: bytes read per function, reject on
an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
the held-out guard
- metrics.py precision (PPV), false alarms per 100 pushes, exploit-verified
rate, and the plain headline, layered on score.py
36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lead with the names a security engineer already knows instead of the
coined ones:
- catch rate -> recall (true-positive rate)
- false-alarm rate -> false-positive rate (still reported as false
alarms per 100 pushes)
- proved-catch / proof rate -> exploit-verified findings (a working
proof of concept)
- precision stays precision (positive predictive value)
The plain-English question for each metric stays as the explanation.
Updated the valid-benchmark brief and the roadmap. No numbers changed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran the humanizer rules over 79 prose files in the active specs (brief,
design, requirements, research and narrative docs). Removed em dashes,
significance inflation, forced rule-of-three, bold-label list items and
filler. Skipped tasks.md checklists and the retired specs (harnessx-*,
openrouter-sast-harness, unseen-repo-evaluation).
Every number, path, command, hash and identifier was preserved: checked
each file against HEAD for drift in numbers, inline code and code fences,
zero drift across all 79 files.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in
the wrong place about half the time, and ~76% of entries are library-internal (not
reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise.
Retired as a metric and a gate across the roadmap and briefs; no pool-scored number
is quoted anywhere.
Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the
fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified;
ordinary negatives from the same repos; unseen repos; the whole tool measured; plain
metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch).
Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two
active briefs via Codex, then reviewed the diff: only these files changed, every
number/path/hash/command is unchanged, and the retired-metric decision is intact.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Follow-up to a36f99e (the prior git-add aborted on the already-removed pathspec and
left this edit unstaged). Removes the change-detector image assertions and renames
the function to test_rustfs_engine_queue_policy; the least-privilege rustfs policy
assertions are what remains.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
They pinned exact Dockerfile text / an engine-stage SHA, so every legitimate image
edit broke them and the "fix" was to paste the new value back — a change-detector,
not a contract. Deleted:
- tests/test_unseen_image.py (test_image_stages) entirely.
- the Dockerfile-text assertions in test_engine_queue's image test.
Kept as a genuine contract: the rustfs least-privilege queue policy assertions
(renamed test_rustfs_engine_queue_policy). The partial-clone extraction test and
its new hermetic-git-env fixture stay — that one verifies real behaviour.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- test_engine_image_and_rustfs_policy: root Dockerfile was refactored to the
digest-pinned ousast-task-base (carries semantic+s3 deps) + a joern-build stage,
installing the package --no-deps. Updated the stale `COPY benchmarks` (now
--chown=1000) and `.[semantic,s3]` (now --no-deps + joern-cli copy + pinned base)
assertions to the new contract.
- test_image_stages: plane/Dockerfile.engine-task refactored the same way; the
split marker is now `FROM ${ENGINE_REF} AS replay`, base is ousast-task-base
(not ax-task-runner:v0.3.1), and the replay stage no longer installs git (base
provides it). Re-pinned the engine-stage SHA and updated the markers/assertions.
- test_partial_clone_lazy_fetch: passed alone but failed mid-suite — a leaked
GIT_DIR/GIT_OBJECT_DIRECTORY/alternates from an earlier test changed the
missing-blob count. Added an autouse fixture that neutralises inherited git
control vars so every test in the file is self-contained (reproduced + verified).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roadmap "Where we are" table said Baseline (G0.2) = "Not yet, run on kube" -- stale.
Record 07 (G0 exit) was completed right after on the docker lane (e901640, 1300/1300,
catch 3.3% / FP 3.1%, regex/quick-rule floor, engine_covered=0). G0 is exited; the kube
`ax` lane was never needed for the baseline. Reframed the readiness plan: steps 1-4 done,
the `ax` lane moves to G1 (engine arm at scale, gated by spec tasks 4.4/5.4).
Also fixed the ax binary in the coordinator commands: use $HOME/go/bin/ax, NOT
$HOME/.local/ax-v0.3.1/ax (the latter hits the AX-lane dispatch blocker recorded in the
unseen spec tasks.md). README.md + benchmarks/unseen/README.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
test_no_pair_repeats_a_repository_the_benchmarks_already_name failed: advisory-fix
catalog repos also appeared elsewhere under benchmarks/. Root cause split two ways:
1. Genuine leak: 16 catalog repos are also subjects in benchmarks/search/private/
pilot.json (a real private pilot population). Removed those 16 pairs from the
three catalogs and their verification.json rows (totals recomputed, catalog ==
verification preserved).
2. Spurious: the overlap scan greped ~49k vendored files under
benchmarks/search/private/runs/*/checkout/node_modules/, so any catalog repo a
vendored dependency's README names collided. Added a narrow vendored-tree skip
(node_modules/vendor) to the scan rather than deleting labelled pairs over a
dependency mention.
Implemented from Codex's first pass (gpt-6.1-sol), then gated and corrected: Codex
removed 20 pairs; 4 of those (backstage/backstage, koajs/koa, beaugunderson/
ip-address, cakephp/cakephp) were driven only by vendored/non-population mentions
and are kept. Also excluded advisory-fixes-spread.json (catalog-derived report)
and research-2026-09-26.md (selection write-up) as non-population.
tests/test_advisory_fix_pairs.py + tests/test_learn_labels.py: 53 passed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`pre-push-for-developers.md` and `use-case-fix-and-reverify.md` (authored
2026-10-07) were in docs/ but absent from nav, so `mkdocs build --strict`
errored and test_docs_site::test_site_builds_strictly failed. File both under
"Using it", restoring the strict build.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Call-graph trace: no shipped product path or in-repo benchmark/test reached the
real single-query JoernBackend.query("taint"). The push/scan/replay path is
double-shielded -- batch fetch (run_batch) in Phase 2, and a _prefetched replay
CpgResult in Phase 3 whose run ignores the query -- so scan_region's cpg.run
("taint") never reaches JoernBackend.query. Benchmarks use the batch path; tests
use fake canned-row results. Only out-of-product ad-hoc harnesses hit it.
Regression onset 562b670 (2026-09-25).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
taint.request_params emits guards/fixedOrigin/originAnchors/quotedSanitizers/
fieldParameterSources, which taint.sc reads from per-request JSON in its BATCH
branch but does NOT declare on `@main def exec` and does NOT consume in its
single-request branch (it defaults them). The single-query path passed them as
`--param guards=...` (disposable) or named args (session), so joern errored
`Unknown arguments` and every cpg.run("taint", ...) failed -- a silent zero of
the shape AGENTS.md warns about. scan_region->taint_verdict->_usable_flows and
the pair-path verdict()/verdicts() were dead; the push/scan batch route was
unaffected (it carries the extras as JSON the script parses itself).
Fix: parse the script's `@main def exec(...)` signature once (_declared_exec_params
+ memoised _declared_params) and, in query(), filter params to the declared set
before building either route. Faithful, not lossy -- the single-request branch
could never use those 5 fields. No-op for config/dominance (their request_params
emit only declared params; verified). Only taint sheds the 5 extras.
Verified end-to-end on banks (patched backend in the replay image): single-query
path now runs; baseline 0 findings, annotated search 2 path/model_entailed
findings (prompt->write_text). Regression test asserts no undeclared param
reaches the command line and the query parses, with a fake runner (no Joern).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.
Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 hits have a runtime oracle (search FAMILY_ORACLES: injection->sql/command, ssrf for
untrusted_destination -- corrects the earlier families.toml-based 'dcmcp unprovable'). But
search/verify.verify() needs an AUTHORED runnable demo + both head/base building OFFLINE + a
sandbox differential (3/3 affected vs 0/3 safe). The scan/push path reaches only suspicion/
static_corroboration; no code path from a scan suspicion to exploit_demonstrated, no Coordinator
caller from the push flow. So proof-arm lift AT PUSH TIME = 0/3 -- proving is a per-target,
offline-build, runtime, differential operation that can't run inside a pre-push hook (prior
per-pair demos were mostly inconclusive on this wiring).
Strategic consequence: proof/BLOCK is an ASYNC lane (CI/plane/triage with authored demos), not
pre-push. Honest pre-push product = advisory-with-conditions (bet #2); proof arm = async BLOCK
lane (bet #3). Local sandbox exists (userns).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran ousast scan --mode deep on the 5 app-shaped + sound pool targets (the fair change-scoped
eval set from the shape classification). Detection 3/5: graphiti (hunter, Cypher injection),
mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE overlay -- 'outbound.request reached
from parameter' on readFileFromUrl; parameter-as-source fired because an entry point is present).
2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families families.toml
documents as limited.
This validates the strategic pivot: the near-zero catch was substantially a TARGET MISMATCH --
on app-shaped repos the engine itself detects (0 -> 3/5). But all hits stop at suspicion/
static_corroboration; release-grade BLOCK still needs proof-arm elevation + family coverage +
the DSPy hint loop. Cost ~$0.007/repo. So bet #1 (fix eval target) is necessary and material
but not sufficient.
Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat,
DjanGoat, vibe-py); measure proof-arm suspicion->proven lift.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Named the core problem with the maintainer: the tool spots dangerous ops but cannot prove
attacker-reachability -- the trust boundary lives outside the changed library code. Chosen
strategic bet (of three): FIX THE EVALUATION TARGET FIRST.
LLM shape classification of the 49 public pool vulns ($0.033): reachability_in_repo only
24% (12/49); 76% are pure libraries. A legitimate change-scoped catch needs sound label AND
reachability in-repo -> just 5/49 (~10%): graphiti, desktopcommandermcp, malla,
document-merge-service, mcp-remote. Shape predicted the hunter outcome (graphiti app-shaped ->
caught; bedrock library -> missed). So record-07's 3/92 is scored against a ~90%-illegitimate
denominator -- the near-zero catch is largely a target mismatch.
Next: deep scan on the 5 fair targets for a per-tier catch rate; then an app-shaped eval set
from known-vulnerable apps. Artifacts: 2026-10-12-pool-p1-shape-classification/ (+ earlier
label-audit, hunter-sound-targets). Memory: evaluation-target-mismatch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed:
- graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces
build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the
labelled function, location-matches). The static engine produced zero findings here.
- bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never
named the install_packages -> pip command injection.
Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything
stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs
proven findings.
Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO,
pluggable; evolving field) whose objective is to learn what hints/annotations steer the model
to the real sink, sourced from the framework knowledge already shipped (no new user
annotation), with LLM-proposed boundaries graduating into the facts.
Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen
the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
