CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.
Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.
Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 90.0%
leak 12.3% 6.7%
detection -- 96.7%
cost ~$0.69 $0.0068
27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).
One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.
The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
5.1 inverts what the corpus is for. Previously a pair the detector missed was a
DETECTOR failure, to be averaged away over K runs and chased with a prompt
change. The arbiter is deterministic: it will miss that pair the same way every
time, and the only thing that moves it is better modelling. So a pair the model
cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every
gap carries one even when the caller supplies none -- a bare gap teaches
nothing.
`covered` means the model reached a strictly higher rung on the vulnerable side
than on its fix. Flagging both sides equally is not coverage: it distinguishes
nothing, which is exactly what group 2 measured taint reachability doing on
every parameterised SQL pair. The harvest seam takes the fix commit and derives
the vulnerable side as its parent, with the fix diff as the oracle -- nothing
hand-labelled.
5.2 keeps three disciplines from the deleted publish layer, each because its
absence caused a real error here:
* per-slice rows always -- `render` RAISES without them. A mixed headline
once hid agent-written code scoring 44% against the development slice's 83%.
* the overfitting gap beside the headline, and ABSENT rather than zero when a
slice is missing: reading an unmeasured slice as "no overfitting" is the
flattering failure, not a neutral one.
* the rung on every finding, so a suspicion cannot be read as a result.
Run end to end against the committed ceiling artifact, the report says things
worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an
overfitting gap of +16%, and gaps grouped by family with their reasons --
injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no
verdict), untrusted_destination 5 (4 entailed on BOTH sides).
713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
A full re-measure with the closure-scoping fix is in flight; these numbers are
the pre-fix baseline and will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
