From a762860b177af1684132f9db4dd0b5233b3529de Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 10 Sep 2026 07:49:35 +0200 Subject: [PATCH] docs(contributor-scan): the 637-file premise was wrong, and it was my own shim This task opened with "php2cpg builds a 4.3MB CPG for a 637-file plugin and importing it throws". It never read the plugin. `php` on this machine is a local shim that runs php:8.3-cli in docker, mounting $HOME/joern, /tmp and $PWD -- and not $HOME/.cache, which is where every pinned checkout lives. PHP could not see one file of it: php -r 'var_dump(file_exists($f));' -> bool(false) # file plainly there php2cpg <637-file checkout> -> 6,224-byte CPG, exit 0, ZERO reported failures Every parser batch failed with `File does not exist.` and php2cpg wrote an empty graph. Second false conclusion this same shim has manufactured -- 5.5 is the first, recorded there as "NOT A DEFECT, it was the shim" -- and the second time a measurement here was taken through a broken instrument. Local only: the shipped image installs php-cli natively, so no released path was affected. And the engine's own honesty machinery was right all along -- a scan over that graph reports `cpg_empty`. It was the hand-run php2cpg and the note in this task that read a wrong conclusion out of it. With the cache mounted, the actual envelope for 637 files / 117k lines: wall 33.0s peak RSS 1,217 MB CPG 1.6 MB 156 drops (69 files) Affordable on a CI runner rather than a vertical-scale box, which was the open question behind this task. The drops are per-BATCH, not per-file. php2cpg hands its parser ~20 files per invocation and a failed batch reports every file in it; dropped file sizes run from 67 bytes to 433 KB, so size has nothing to do with which files are lost. Being in an unlucky batch does. That is why retrying works, and why "exclude the difficult file" was the wrong mental model -- worth correcting in the sharding rationale once the whole-repository scan finishes measuring. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/contributor-scan/tasks.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 0fc75ff..e7b32e0 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -609,7 +609,32 @@ - The excluded-file tree is symlinks at the ORIGINAL relative paths, because a region asks about `classes/class.memberorder.php` and the queries match a filename by suffix; a flattened copy would answer about a file nobody asked about. - - Still open: nothing here recovers a cross-shard flow, and the 637-file build remains unmeasured. + - **The 637-file premise was WRONG, and it was my own instrument. 2026-09-10.** This task opened with + "`php2cpg` builds a 4.3MB CPG for a 637-file plugin ... importing it throws". It never read the plugin. + `php` on this machine is a local shim that runs `php:8.3-cli` in docker, mounting `$HOME/joern`, `/tmp` + and `$PWD` -- and NOT `$HOME/.cache`, which is where every pinned checkout lives. So PHP could not see a + single file: + + php -r 'var_dump(file_exists($f));' -> bool(false) # on a file plainly there + php2cpg <637-file checkout> -> 6,224-byte CPG, exit 0, ZERO reported failures + + Every parser batch failed with `File does not exist.` and php2cpg wrote an empty graph. That is the + SECOND false conclusion this shim has manufactured -- see 5.5, "NOT A DEFECT, it was the shim" -- and + the second time a measurement here was taken through a broken instrument. + - It is local only. The shipped image installs `php-cli` natively (`Dockerfile`), so no released path was + ever affected, and our own honesty machinery was right the whole time: a scan over that graph reports + `cpg_empty`. It was the hand-run `php2cpg` and the note above that drew the wrong conclusion from it. + - **With the cache mounted, the real envelope, one frontend build of 637 files / 117k lines:** + + wall 33.0s peak RSS 1,217 MB CPG 1.6 MB 156 file-drops (69 distinct files) + + Affordable on a CI runner rather than a big machine, which was the open question. + - The drops are per-BATCH, not per-file: php2cpg gives its parser ~20 files per invocation and a failed + batch reports every file in it. Dropped file sizes run from **67 bytes to 433 KB**, so size has nothing + to do with which files are lost -- being in an unlucky batch does. That is why retrying works, and why + "exclude the difficult file" was the wrong mental model. + - Still open: nothing here recovers a cross-shard flow, and the whole-repository SCAN (not just the build) + is still measuring. - Still open: wiring that exclusion loop, the 637-file build with it in place, and whether the REST handler two calls away can reach the sink -- the slice found it through a same-function flow, not across the object.