Open_Ultra_SAST/.kiro/specs/zero-with-a-reason/brief.md

Brief: zero-with-a-reason — a finding count is meaningless without a denominator

Status 2026-10-02: dormant. Brief only and unapproved; not touched since 2026-09-10.

Status: brief only, unapproved.

Why now

"can you think of an architecture to avoid these 0 findings issue?"

Because it is not one bug. It is the same bug seven times, and every single one was caught by a human noticing that a number they had a reason to expect was zero:

#	what produced the zero	how it was caught
1	CpgResult had no run_batch, so batching was unreachable	a scan took 275s and the reason did not add up
2	a 1.49MB batch over execve's 128KB argv cap → E2BIG → {}	500 regions "examined" in 0.05s of query
3	a 117k-line CPG threw during overlay load → {}	same shape, found again by hand
4	the budget's break halted unpaid CPG work	6 of 22 VAmPI regions examined
5	protected function did not match ^\s*function → one file-level region	a scan of a known-vulnerable file went quiet
6	php2cpg dropped every file and exited 0	64 regions, 0 findings, 0 degradations, over an empty graph
7	a Joern [INFO ] line landed inside the fenced payload	205 requests lost in one batch

Each was fixed at its own seam, and each fix was correct. None of them could have been found by the tool. The pattern is that the pipeline reports the last number in the chain and discards every number before it, so all seven look identical from outside: 0 findings.

The rung ladder cannot help here by construction. It labels verdicts, and the failure is that no verdict was produced. There is nothing to put on a ladder.

The idea

Every zero must be decomposable, and every scan must be able to prove it did work. Three mechanisms, independent, in order of value per line of code.

1. The funnel — make the denominator first-class

The scan is already a chain of narrowing stages, each of which computes a count and throws it away:

files discovered        637
files parsed into graph   12      <- 625 dropped
methods in graph         153
regions mapped            64
questions asked          205
sink sites matched         0      <- the zero is HERE
flows considered           0
verdicts                   0
findings                   0

The rule: a zero at stage N is legitimate only if stage N-1 is non-zero. Otherwise the zero belongs to the earlier stage and is reported there. All seven bugs above are a different row of that table, and the table distinguishes them without knowing anything about what went wrong.

Cheap, because every number already exists at the moment it is computed. ModelScanResult grows a funnel; the coverage disclosure (Req 5.6, already written) is where it prints. The hard rule that gives it teeth: a report may not describe a repository as clean without the funnel beside it.

2. The dumb oracle — an independent lower bound on the graph

The CPG is an elaborate instrument. A regex over the source text is a stupid one, and stupid instruments fail differently — which is the entire reason to have two.

Before querying, count the fact tables' sink names in the raw source text. That is a floor on what the graph must contain:

source text contains 26 occurrences of $wpdb->{get_var,query,...}
the CPG's sink match returns 0
-> degradation: graph_below_text_floor, both numbers reported

This catches an empty graph, the wrong frontend, dropped files, a broken name matcher, and a missing overlay — bugs 2, 3, 5, 6 and 7 above, none of which it needs to know about in advance. It is the same trick the pair corpus already uses: two independent measurements of one thing must agree.

What it cannot do: say anything about over-reporting, or about flows. It is purely "does the graph contain the syntax the text contains". That narrowness is why it is trustworthy.

3. The canary — an end-to-end positive control

One synthetic file with an unmistakable flow, one extra request in the batch:

<?php function __ousast_canary() { global $wpdb; $wpdb->query("SELECT " . $_GET['c']); }

If the canary is not entailed, the run is invalid, not clean. That single assertion proves the frontend ran, the file reached the graph, overlays applied, the fact tables loaded, the query compiled, the batch round-tripped, the payload parsed, the arbiter ran and the ladder assigned a rung.

With one honest caveat that must be designed around: adding a file to a PHP build can itself trigger the php2cpg race — a two-line file added to class.memberorder.php is exactly what emptied that graph. So on PHP the canary belongs in its own build (ousast doctor), where it proves the toolchain works, and the funnel and the oracle prove that this build worked. They are complementary, not redundant.

3b. The instrument must prove it can read the input

Every instrument failure in this project has the same shape, and it is not a coincidence: the input was unreadable and the tool reported a plausible zero. file_exists() returns false, the parser writes nothing, the frontend exits 0, and "0 files dropped, 5,703-byte graph" reads exactly like success.

It has now happened four times in one day — a docker shim that did not mount ~/.cache (twice, hours apart, the second time after I had diagnosed the first), a measurement harness whose script path did not resolve from its working directory, and a container that could not follow the symlinks in the tree it was handed.

The rule that would have caught all four costs one line: before measuring, make the tool open one input and fail loudly if it cannot. Not a check of the output, which is where all the effort has gone -- a check that the input arrived. _graph_is_complete is this idea applied after the fact; the cheaper version is to ask first.

This applies to our own measurement scripts as much as to the engine. A harness that prints 0.5s and rows=? where a JVM takes fifteen seconds to start is not reporting a fast query, and it should say so in those words rather than leaving a question mark for a tired reader to skip.

3c. Nothing compiles the CPG queries

taint.sc, dominance.sc and config.sc are 900 lines of Scala that the test suite never compiles. A signature change to rowsFor shipped with 850 tests passing and broke every query at runtime; it was found by a measurement failing, not by CI.

The canary in mechanism 3 covers this exactly -- a query that cannot compile cannot entail the canary -- which is another reason it belongs in ousast doctor rather than in-band.

4. Types that cannot express the confusion

dict | None meaning "no rows" versus "could not ask" is a convention, and bugs 2, 3 and 7 are each a place where the convention was silently violated at a boundary. A two-case Answer (Answered(rows) / Unavailable(reason)) makes mypy force the distinction at every call site. Mechanical, small, and it prevents recurrence rather than detecting it.

What this is not, and must not become

• Not a confidence score. The rung ladder exists precisely so uncertainty is carried as a claim about evidence rather than a number nobody can act on. A "coverage percentage" that averages the funnel away would undo it.
• Not a retry loop. Retrying a failed build until it succeeds hides bug 6's race instead of reporting it. Three of five builds succeeding is a fact about the toolchain and belongs in the output.
• Not anomaly detection over historical finding counts. 6.1's baselines already give the diff, and "findings dropped 90%" is a lagging indicator that needs a known-good prior run — which is exactly what a first scan of a new repository does not have.
• Not more code than it saves. The funnel is plumbing that already exists, the oracle is one pass over files the preprocessor already reads, the canary is one file and one request.

Open questions for the design phase

1. Does the funnel live on ModelScanResult, or is it a separate ledger the driver writes into? The stages span preprocess, mapping, regions and scan, and only the last of those returns a result object.
2. Is the text floor a hard degradation or a hard failure? A file the frontend legitimately excludes would trip it, so the threshold needs stating rather than assuming zero tolerance.
3. Does the canary belong in ousast doctor for every language, or in-band for the ones where adding a file is safe? The PHP answer is settled; Python and C are not.
4. Which of the seven bugs above would each mechanism have caught? That table is the acceptance test for this spec, and it can be written before any code is.