CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.
Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.
Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 90.0%
leak 12.3% 6.7%
detection -- 96.7%
cost ~$0.69 $0.0068
27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).
One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.
The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
vampi-users-update-password scored both_silent while the arbiter returned
model_corroborated on the vulnerable side and nothing on the fixed one -- so the
finding existed and the pipeline threw it away.
The residual question was written for the ENTAILED case, but entailed returns
before the judge is ever consulted, so the question is only ever asked for
CORROBORATED. It asserted "in a file where sibling handlers are guarded" and
asked about "the guard its siblings have", printed directly above a witness
reading "and no guarded sibling" -- because corroborated means precisely that
no sibling is guarded. The judge was handed a prompt contradicting its own
evidence line and reasonably answered no.
The question now describes what corroborated actually is: an obligated
operation no guard governs, with no sibling evidence either way, so the model
cannot tell from consistency alone whether the guard is missing or the module
is deliberately public. That is the real open question and the one worth asking.
Verified end to end against a live endpoint: vuln model_corroborated, fixed
silent, the pair splits.
The taint and config questions were checked for the same defect and are
correct -- each describes the corroborated case it is actually asked about.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two things, and a finding that says neither fixes the pairs that prompted them.
The settings table was Flask/FastAPI-shaped (CORS, set_cookie, app.run) while
the corpus contains aiohttp and Django applications, so three config_secrets
pairs contained no modelled security setting at all. Added aiohttp's
session_setup and cookie storages and Django's SECURE_* / DEBUG settings: 8
calls to 24.
And config_secrets spans three abstractions while only one was wired -- a
permissive setting VALUE. Weak algorithm CHOICE (CWE-326/327) is a different
thing: the danger is the algorithm NAMED, not a flag left open. Its data has
been sitting on the sink facts as `weak_literals` since before this feature and
nothing read it. Both abstractions read the same literal, so the arm is small:
hashlib.new("md5") is now entailed and hashlib.new("sha256") is not.
The finding: the three silent config pairs are CORPUS DEFECTS, not model gaps.
Measured similarity 23%, 45% and 53% -- dsvpwa-dsvpwa-main pairs `def main()`
against `def run(self, handler)`, a different function entirely. No settings
table can make those split, and it is worth saying so rather than letting the
work imply otherwise. The table gap was real and worth closing on its own
merits; it just is not why those pairs are silent.
734 passing, mypy/ruff clean, gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The corrected re-measure returned numbers IDENTICAL to the buggy one, which was
the clue. Routing was fixed, the dominance arbiter was reached, and the finding
was discarded anyway -- because every corroboration was put to the judge as a
TAINT question: "a sanitizer appears on the path: yes, does it suffice?".
For a missing authorization guard that question is meaningless. The judge
answers no to something that does not apply and the claim is dropped. All three
vibe-py access_control pairs scored both_silent this way even though the
dominance arbiter had returned model_corroborated for each of them, which I had
verified by hand earlier in the session.
Fourth instance in this feature of one shape: taint-specific behaviour applied
to every family. Sanitizer matching, config routing, family dispatch, now the
residual question. Each was exposed by a measurement rather than by a check.
The question is now phrased for the arbiter that raised it -- a guard asymmetry
for dominance, a permissive setting for constant abstraction, sanitizer
sufficiency only for taint.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
scan_region took a TaintSpec and called taint_verdict, so access_control and
config_secrets -- whose dominance and constant-abstraction arbiters have worked
since group 4 -- had no pipeline path at all. On the first end-to-end vibe-py
run every one of their pairs scored `both_silent`, which reads as the model
failing when in fact no code reached the arbiter.
This is the same class as the ceiling harness routing config_secrets through
taint: family routing fixed in one place and not carried to the other. Third
time this shape has appeared in this feature, which is worth saying plainly.
Dispatch is on the spec TYPE rather than a family name, so a family without an
arbiter cannot be silently routed to the wrong one -- it simply has no spec and
is skipped, visibly.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Everything before this built an ARBITER: something that judges a flow taint
reachability had already found unaided. Nothing proposed to it. That is why its
recall was never comparable to a detector's, and why comparing 41.3% entailment
against the LLM hunter's 76.7% pair-correct was comparing two different kinds
of thing. model/judge.py and enumerate_candidates were called by no production
code at all; every number so far came from a measurement harness.
The wiring turns on a distinction judge() alone conflated:
the model CONTRADICTS it has coverage at this site and says no -- a
sanitizer on the path, or a declared-safe sink shape.
The claim is dropped (Req 8.2).
the model CANNOT DECIDE no modelled sink or source reaches the site at all.
The model is SILENT, not negative, so the claim is
reported at `suspicion` (Req 8.4).
Treating silence as refutation is the tempting mistake and would be wrong here:
23 of the 39 misses in the entailment ceiling were sinks simply absent from the
fact tables, so "no flow" usually means the model cannot see it rather than
that nothing is there. The rung carries that uncertainty on the finding's face
instead of a confidence score nobody can act on.
An entailed site still costs zero model calls. The candidate budget is bounded
so a dense region cannot become an unbounded spend.
Smoke-tested end to end against a live endpoint: 2 pairs, detection 100%,
$0.0024. The full vibe-py measurement -- the first like-for-like comparison
with the hunter's 76.7% pair-correct and 12.3% leak -- is running.
727 passing, mypy/ruff clean, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
