Four instrument failures in one day, all the same shape: the input was
unreadable and the tool reported a plausible zero. file_exists() false, parser
writes nothing, frontend exits 0, and "0 dropped, 5,703-byte graph" reads
exactly like success.
* a docker php shim not mounting ~/.cache -- twice, hours apart, the second
time after I had already diagnosed the first
* a measurement harness whose script path did not resolve from its cwd,
which printed 0.5s and `rows=?` where a JVM takes 15s just to start
* a container that could not follow the symlinks in the tree it was given
The rule that catches all four is one line and runs before the work: make the
tool open one input and fail loudly if it cannot. Every mechanism in this brief
so far checks the OUTPUT; this checks that the input arrived.
Also records that nothing compiles the CPG queries. A signature change to
rowsFor shipped with 850 tests passing and broke every query at runtime -- found
by a measurement failing, not by CI. The canary already proposed here covers it,
since a query that cannot compile cannot entail the canary.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Seven separate defects in this project have produced "0 findings" for reasons
that had nothing to do with the code being scanned: an unreachable batch, an
argv cap, a CPG that threw on load, a budget that halted unpaid work, a regex
that missed `protected function`, a frontend that dropped every file and
exited 0, and a Joern log line inside the fenced payload.
Every one was fixed at its own seam and every fix was correct. Not one could
have been found by the tool, because the pipeline reports the last number in
the chain and discards every number before it -- so all seven look identical
from outside.
The rung ladder cannot help by construction: it labels verdicts, and the
failure is that no verdict was produced.
Proposes three independent mechanisms and one refactor:
* the funnel -- files discovered, files parsed, methods, regions, questions,
sink sites, flows, verdicts, findings -- with the rule that a zero at one
stage is legitimate only if the previous stage is non-zero. Each of the
seven bugs is a different row of that table.
* a text floor: count the fact tables' sink names in the raw source, and
degrade when the graph contains fewer. Two independent instruments, the
dumber one bounding the clever one.
* a canary flow that must come back entailed or the run is invalid -- with
the caveat that on PHP, adding a file can itself trigger the php2cpg race,
so it belongs in `ousast doctor` rather than in-band.
* an Answer union so "no rows" and "could not ask" cannot share a value.
And what it must not become: a confidence score, a retry loop, anomaly
detection over historical counts, or more code than it saves.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
