CI runs ruff format --check and ruff check over the whole repository; 29 files had drifted from the formatter and
four measurement scripts carried lint findings, so every push since 25ae4b4 failed before pytest ran.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sweep over the pilot's hunter records: most 'vulnerable' suspicions on ordinary
pushes sit far from the diff. Scope to the change, then confidence, then the
config_secrets family; 5/12 recall kept at 8.8 false alarms per 100 (in-sample).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Arm (b) hunter over all 252 changes: recall 6/12 raw, 5/12 with the classifier
filter; false alarms 92.5 and 54.4 per 100 ordinary pushes. Arm (a) push hook at
163/252: recall 0/12, 0.7 false alarms per 100. Per-positive table, classifier
stats, the one hook false alarm, and the disk fix for the replay output.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run of the valid detection benchmark. Labels come from the fix
site (fix^1), never SZZ.
- 32 held-out Python/JS advisories sourced from the public pool-p1 set,
CWE + text from OSV. 133 candidate locations -> 125 after dropping
tests/minified.
- 125 DeepSeek verification calls: 31 locations passed all three checks
(vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos.
- Maintainer spot-check accepted all 12.
- 20 ordinary negatives per positive change from the same repos, 240 total.
- Held-out guard: 0 dropped against the training-only used set.
- Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and
NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/.
Three fixes found by the live run:
- catalog.py: skip a site the fix renamed or removed instead of aborting.
- assemble.py: negative ratio counts positive changes, not accepted
functions (malla would have demanded 260 negatives). Test updated.
- held-out guard: run against training sources only; build_used() also
scans measurements/populations, which flagged the 12 repos via my own
SZZ-pool analysis outputs (circular).
Replay and metrics not yet run; see NOTES.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
