Marius CiepluchandClaude Opus 4.8
a43e6293f5
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek (advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%) sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7, output_encoding 1/8, config_secrets 1/4. Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES). Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine missed both = the real detection gaps for the LLM-guided hunter. Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/. Co-Authored-By: Claude Opus 4.8 <[email protected]>
3.3 KiB
pool-p1 label audit (LLM corpus-honesty pass) — 2026-10-12
Why
The engine-on pilot's banks "miss" turned out to be a mislocated SZZ label: the pool labelled the vuln at
function=_load, head_lines=[[30],[104],[138-141]], but the real sink is from_prompt_path line 44
(outside the changed lines) and it pre-exists in base — so a change-scoped pre-push hook correctly stays
quiet. The pool documents this bias itself ("SZZ last-writer proxy… can misidentify"; "the flaw may already exist
in base, so a delta hook can correctly stay quiet"). We widened the check across the whole public pool to see if it
is systematic (maintainer directive 2026-10-12, "widen via LLM").
Method
For each of the 49 public pool vulns (the 51 private/redacted entries have no code to audit), one DeepSeek
deepseek-flash call given: the GHSA advisory (ground truth) + the vuln file at the head revision + the pool's
label (function, function_lines, changed head_lines). The model returns JSON: the real sink location, whether it
falls inside the changed lines (±10), introduced-vs-preexisting, and label_sound (fair target = sink in the
changed lines AND change-introduced). Audit cost: $0.038 total (~$0.001/call), ~2s/call. Raw per-repo verdicts
in results.jsonl; aggregate in summary.txt. Harness: scratchpad label_audit.py.
Result
- label_sound = 23/49 (47%); unsound = 26/49 (53%).
- introduced 23 / preexisting 25 / unclear 1; sink_in_changed_lines True 33 / False 16.
- By family (sound/total): injection 10/13, access_control 5/8, untrusted_destination 4/7, deserialization 1/2, config_secrets 1/4, path 1/7, output_encoding 1/8. Injection labels are mostly sound; path and output_encoding labels mostly mislocate or pre-exist.
Implications
- record 07's floor (catch 3/92) is scored against a corpus ~half of which is unfair to a change-scoped hook. On the ~53% pre-existing/mislocated vulns a delta hook should stay quiet; counting them as required catches depresses the measured catch rate. The honest catchable denominator is the sound subset. record 07 should carry this caveat and, ideally, a sound-subset recomputation (public repos only locally; private entries un-auditable).
- Fair pilot re-score: of the 5 engine-on pilot repos only bedrock + graphiti (injection) are sound, and the engine missed both — those are the real detection gaps to attack with the LLM-guided hunter. banks, launch-editor, socketio are unsound (quiet is defensible).
- The 23 sound targets are the fair local evaluation set for the "#2 guided by #1" detection work.
Caveats
- The audit is itself an LLM judgement; validated against the manual banks finding (matched). 5
sound=Trueverdicts are confidence <0.7 and warrant a human spot-check: jupyter-lsp, role-strategy-plugin, codeigniter-restserver, graphiti, mcp-remote (graphiti is a pilot target — its soundness is medium-confidence). - Advisory line references are sometimes approximate; the model estimates the sink line.
- Private pool entries (51/100) are not audited — no code. The public-subset rate may not transfer to them.
- deepseek-flash is a general model, not security-specialised; a GEPA-style reflective-retry pass (future spec) and the GLM "cyber" model (future spec) would raise audit confidence.