The warning once read 22126 of 22125 requests answered in parts, because the census rides the
answer under its own key and was counted as a request. An instrument that miscounts by one casts
doubt on every number beside it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A hard per-invocation ceiling is all-or-nothing. A batch killed at 301 s lost every answer it had
already computed, and the retry paid a fresh JVM start per part to ask them all again: measured, 116
requests killed at 300 s, then eight restarts carrying the same everywhere-sinks in every part.
Three coordinated changes make portioning, not the cap, the thing that governs cost.
The taint query STREAMS. The census first, then one line per request as it is answered, flushed as
it goes, inside the unchanged fence. The other queries do not stream and need not; the driver
reads both forms, and a legacy single document parses exactly as before.
The driver HARVESTS a kill. The process group is terminated first -- a first cut read the pipe
before killing and a pipe whose writer is alive never reaches end-of-file; the suite own
hanging-engine control sat on that read for twenty-eight minutes -- then the pipe is drained and
every complete line becomes an answer, marked partial. The retry asks only the remainder.
Portion sizing ADAPTS. After each portion the observed seconds per sink visit, less the fixed
start, resizes the rest toward a 120 s target; a portion that did not answer halves the budget
rather than being fitted, because its true cost is unknown and above the ceiling. A portion cut
down to fit puts its remainder back at the front of the queue.
A census LINE carries nothing but the census. A first cut misread a legacy document that carried the
census beside its answers as a census line and dropped every answer it sat beside; a control caught
it. Task 7.8 in contributor-scan.
Verified: 1,423 host tests pass, nineteen skipped, including new controls for the assembler in both
forms and under a kill, for a retry that asks only the remainder, and for the sizer. Not yet
verified: the joern-gated streaming check and the plugin measurement, which wait for the transfer
replay to free the machine and are stated as open in the task.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two different timeouts reached the same handler and got the same consequence. If the SCAN deadline is
gone, cancellation is right: stop everything promptly, which is what the push two-second gate
measures. If a single query merely exceeded its own per-query ceiling, the scan may have most of its
budget left and every other batch is independent and cheaper -- and cancelling took them all.
Measured on Paid Memberships Pro: one batch of 116 requests hit a 300 s ceiling 734 s into a 2,700 s
scan, and the four batches behind it returned in 0.0 s unanswered with thirty-three minutes to
spare. The handler already distinguished the two cases in the reason it recorded, deadline_exhausted
against process_timeout, and then set the cancel deadline in both.
Fourth instance this week of one rule, now written in the design: a failure is owned by the narrowest
scope that caused it.
1,420 tests pass, eighteen skipped, including the cancellation gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine hypotheses about the PHP query cost have now been eliminated by rebuilding the request outside
the driver: batch size, call depth, parameter sources, hook table, argument length, sharding,
dataflow cost, the malformed-CPG bug and the engine version. Every one of them answered in under
four minutes while the driver spent whatever ceiling it was given.
The reason the guessing continued is that a scan issues several invocations and the record could not
say which of them burned the budget. It now logs the query kind, the request count, the elapsed time
and whether it answered, for the whole batch and for each part of a split retry, including the part
where a split is abandoned for want of time.
This is task 16 applied to the backend rather than to a harness: an instrument that cannot name its
own expensive step turns a measurement into a guess, and this one has been a guess since September.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every PHP failure logged `cpg batch taint failed: dure1.java:10)`, which is the
middle of a stack frame and says nothing. A trace puts its exception on the FIRST
line and frames after it, and the backend logged the last 400 characters, so the
cause was discarded and the noise kept. Four investigations were spent on that.
Failures now report the leading non-frame lines, which is where the exception and
its message are, with the frames dropped. A control asserts an OutOfMemoryError
survives while a frame does not, and that a timeout and an empty exit stay
distinguishable.
1,411 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PMPro's 1,274 PHP files produced a response the engine could not serialize: it
failed inside ujson while building its own answer array, and every question in that
batch was lost with it. Three budgets from 900 to 3000 seconds made no difference,
because the size of the answer was the problem rather than the time available.
One invocation stays the first thing tried, so a working scan costs exactly what it
did. Each request in a batch is independent, so a batch that fails as a whole can
still answer in parts: on failure the requests are split and retried, and a part
that fails leaves its own requests unanswered rather than discarding every answer
beside them. Only a batch where no part answered is reported as unavailable, which
keeps "could not ask" distinct from "no rows".
Controls cover all three shapes: a whole batch that fails is answered in parts with
the split recorded, a failing part leaves only its own requests unanswered while the
rest are kept, and a batch no part can answer is still unavailable rather than
empty.
Whether this is enough for PMPro is a separate question the measurement will
answer: a single taint request on that tree already takes 40 seconds and returns
36 KB, so the batch may be slow as well as large.
1,410 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reverts f04a839 on its own measurement. The seven-class profile rerun recorded
joern_parse_census_incomplete on 12 transactions: the faster build was rejected by
its own census gate and the frontend plus overlay pass ran as well, so those
transactions paid two engine starts instead of one. Every workload kept the same
p50 and the verdict stayed NO-GO.
The cause is an asymmetric gate. The frontend path is not census-gated at build
time, so whatever it produces is accepted, while the joern-parse path was gated and
the same shortfall rejected it. Holding the new path to a standard the old one
never has to meet guarantees the fallback fires. The probe tree and the
materialized push snapshot also differ, so equivalence on the checkout did not
transfer to the real path.
What still stands is the underlying timing fact, measured under the production
retention environment on identical graphs, same 44 files, 579 methods, file names,
overlays, Gruntfile and eighteen test files: 11.0 s against 21.0 s. JavaScript does
pay two engine starts where every other language pays one. A future attempt needs a
gate both paths are held to, or a direct comparison against what the frontend would
produce, and must be measured in the push path rather than on a checkout.
The profile also names the remaining cost plainly: a changed-code transaction
spends its budget before any query runs, head build 25.2 s of a 30 s deadline with
query time 0.0 s. Building one new graph per push is the cost, not querying it.
1,401 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JavaScript was the only language paying two engine starts: the declared frontend
and then a separate overlay pass. joern-parse applies the overlays itself. On the
pinned Node tree that is 11.0 s against 21.0 s.
The declared frontend stays as the fallback, because it is what the versioned
retention adaptation and the explicit exclusion policy run through, and the census
decides per build: a graph that does not hold every readable declared file is
rebuilt through the frontend rather than trusted. Exclusions keep the frontend path
outright.
Measured under the production retention environment, both paths produce the same 44
files, the same 579 methods, the same file names, the same five overlays, the
Gruntfile and all eighteen first-party test files, with no file present in one
graph and missing from the other.
My first measurement did not establish that. It ran without the retention
environment, both paths dropped to 25 files, and the apparent equivalence was two
runs discarding the same things. The existing control named for that decision
failed against the change built on it, which is exactly what it is for; the
evidence was wrong, not the control. That control now asserts the intent directly:
the faster path is tried first, and a short census rebuilds through the frontend.
1,401 tests passed, nine skipped. A build saving is not the latency gate; the
profile rerun decides that. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first session-enabled replay produced correct detections and no speedup,
reporting session_query_failed:census:session_request_failed on both sides. A
targeted probe ruled out the sequence I suspected: an overlay followed by a census
in one session both answer correctly, with the census confirming the overlays were
saved.
What the probe exposed instead is a leak. A build runs on a copy of the backend, so
the session started during the build belonged to that copy and was never closed,
leaving a JVM holding the configured 2 GB heap for the life of the process. Two
builds per transaction meant two of them, which is enough to exhaust a 3 GB
container and make every later request fail. It also matches the host running out
of memory during these runs.
The build now closes any session its copy started, and the driver ends the session
with the transaction, since that is what transaction-owned means. Controls cover
both: a build leaves nothing alive on the copy that started it, and a completed
scan closes the backend's session. Closing never fails a scan that found things.
This is a genuine defect found by measurement rather than by the suite, which
passed throughout. Whether it was the whole cause of the census failure is not yet
established; the next session-enabled run will say.
1,399 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The session-enabled NodeGoat run reported session_query_failed:census:
session_request_failed and fell back, which is correct behaviour but says nothing
about why. A bare transport failure cannot distinguish a timeout from a crash from
an out-of-memory kill, and the only account of which it was is in the server's own
log.
A transport failure now carries the exception and the tail of that log, and the
backend logs it with the process exit code alongside its own label. A control
asserts an OutOfMemoryError in the log reaches the caller.
This is diagnosis, not a repair: the run's timings were unchanged because the
session died early and everything fell back to the disposable path.
1,397 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A measured run needs to enable the transport without changing any default. The
switch reads as enabled only for the exact value "1"; unset, empty, "0", "true"
and anything else leave the disposable path in place, and an explicit constructor
argument still wins in both directions.
1,396 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The overlay was the last fixed cost bypassing the session: a whole JVM start,
15.8 s in the measured Node runs, while the census and query batches already route
through it.
`save` writes under the process working directory, which for a session is its own
private scratch rather than the build scratch, so the saved graph is now looked for
in both. Finding neither keeps the raw graph and records it rather than reporting
success, because a graph without its dataflow layer has that layer recomputed by
every later batch, 54 s each on a 637-file plugin, with nothing otherwise saying so.
The disposable invocation remains the fallback and runs unchanged whenever the
session is off or cannot answer.
Controls cover both new paths: an engine that reports success but saves no graph
keeps the raw one and notes it, and a graph saved under the session's workspace is
found and applied.
1,395 tests passed, nine skipped. The transport stays off by default; task 11.5
measures against the frozen profile. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JVM startup, not analysis, is what a push scan spends its time on: each disposable
script invocation costs 13 to 23 seconds while the work takes under a second.
The session owns one Joern server per transaction on loopback with an OS-chosen
port and no published port. Three properties are load-bearing, and the probe that
settled the design demonstrated why each is needed.
An HTTP status is not an answer. The server returns success even for a compile
error, with the diagnostic in its body, so every request redirects the evaluated
code's console output into a fresh private file behind a nonce, and the nonce is
verified before the answer is accepted. The engine's body is retained for exactly
the case where the receipt is missing, without which that failure explains nothing.
A poisoned session answers nothing afterwards, and cancellation kills the process
group rather than the launcher, which this project has already run a machine out of
memory by neglecting.
What the probe established about the engine, after three iterations that each
failed loudly: runScript is absent from the server REPL; a shipped @main script
cannot be nested in a block expression, so definitions go to the top level
unwrapped; and the REPL shadows Console with io.joern.console.Console, which has no
withOut, so capture must qualify scala.Console. With that fixed, both mechanisms
answer identically to the disposable invocation, 0.8 s against 9.8 s.
The backend routes batch and single queries through the session only when
session_transport is enabled, which it is not by default. Any failure falls back to
the disposable invocation rather than reporting a zero, and each script is defined
under its own name because every script declares exec.
Equivalence for taint, dominance and configuration is task 11.2 and gates any
measured run. 1,391 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two decisions the maintainer made, as rows rather than constants. A vendored tree (vendor/, node_modules/,
third_party/) is somebody else's unit: out of the targets in preprocess and out of the graph through the
frontend's --exclude, which the scan hands to the backend because the CPG seam imports only the standard
library. A test path (tests/, test-*.php, *Test.php) stays in scope -- a test that reaches a sink still
names the sink -- but its regions are not shipped and order after the product's. Measured before the row:
207 of WP Statistics's 357 PHP files were vendored, and 14 of the 67 regions ahead of MW WP Form's CVE
were test files. The position benchmark applies both, so it measures the order the scan spends.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
php2cpg drives PHP-Parser through `php` from a script inside its own install. A second install the
containerised interpreter had no mount for parsed 1 of 637 files and exited 0 -- caught by the build
census, and now refused by name before any build. On 4.0.625 with the closure exclusion gated off:
pmpro 637 of 637 in 100 s, WP Statistics 115 s with only the two data tables excluded. Phase-2 v5
verdict and the within-tier reading recorded in the brief.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
joern#6281 is closed as fixed in v4.0.625 (verified here on the reproducer). The version is read off the
jar beside the launcher; below 4.0.625 the miscompiled file is still excluded, at or above it nothing is,
and an install whose version cannot be read keeps the workaround.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
WP Statistics's first region is JavaScript, and a build asked for as JavaScript went looking for
jssrc2cpg, failed both launchers, lost the whole run's output and left a 9.6 MB scratch directory behind.
Also notes upstream: joern#6281 is fixed in v4.0.625.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three checks for three findings that were each silently wrong for a day (task 5.13). A clean build whose
census cannot be taken inside the query timeout is a failed build, by name: every later query would time
out and read as "nothing here", and the launcher fallback no longer rebuilds the same graph without
asking again. The census now carries the graph's saved overlays and the worker JVM's actual heap, and the
backend warns when the dataflow layer is missing (every batch would recompute it) or the heap is not the
configured one (the knob is not reaching the JVM that matters). An empty tree counts as complete, not as
a failed count.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.13, measured on ten real pmpro pairs at callDepth 1 with each join half switched separately and
the 54 s JVM fixed cost subtracted: both halves off 11.1 s/request, field half 10.9 s, hook half past
49 s (timeout). Stage timers put it in `hookSeedsIn("")`: every field access in all 7,061 methods put
through the field join before the first callback was answered. Seeds are now the callback's own body
and file, which is where WP Statistics's `set_current_page` and `$this->rest_hits = ...` both live.
Both halves on: 9.7 s/request. `sourceNodes` is also materialised once per request instead of once
per sink.
The WP Statistics re-verification then exposed the larger term. A `joern --script` run is two JVMs and
`-J-Xmx` sizes only the launcher, so the heap knob governed nothing; the backend now sets
JAVA_TOOL_OPTIONS too. And since PHP builds went through php2cpg directly (1ce7e9d, an hour after the
hook half was verified from a joern-parse build), every query batch has recomputed the dataflow overlay
at load -- 54 s on pmpro, and on WP Statistics an OutOfMemoryError inside OssDataFlow after 21 minutes
at 2 GB, no completion in 25 minutes at 4 GB, and no help from --max-num-def at 1000 or 200. Two
vendored browser-profile tables, one array literal each, were 60% of that graph.
Three changes, all named in the log: `queries/overlay.sc` applies the overlays once at build time and
the saved graph replaces the raw one (joern-parse --overlaysonly NPEs in 4.0.623); a source file over
1 MB is excluded as a data table and reported in `unparsed`; and the seed scoping above. After: the WP
Statistics graph reloads in 8 s, the `record` request that timed out at 520 s answers in 39 s including
the JVM, and CVE-2022-25148 is entailed again with `sourceKind = hook`. A raw-vs-saved payload diff on
pmpro shows identical evidence keys and fewer duplicate flow rows.
Also: the backend's own unit tests left fifteen ousast-cpg-* directories in /tmp per run; conftest now
keeps tempfile under the test's directory. The compile test covers overlay.sc.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Every build allocated a temp directory for the graph, its request files and
joern's own working copy of the CPG. Nothing ever reclaimed it. A day of
scanning left 1,055 directories and 355MB in /tmp on a disk already at 95%,
and the runs that leaked hardest were the ones that FAILED -- which is exactly
backwards, since a failed scan has the least to show for the space it took.
Two mechanisms, because they cover different deaths:
* `CpgResult.cleanup` disposes of the tree, and `scan_repository` calls it
when the graph has answered everything it is going to. Never raises: a scan
that found things must not fail while tidying up.
* A sweep at build time removes trees older than six hours. A process killed
mid-scan never runs its cleanup -- which is how most of the 1,055 died --
so the disposal alone is not enough. Only directories under the system temp
root carrying this module's own prefix are touched, and a scan running in
another process right now is left alone.
Note for the record what was NOT a product leak: 6.3GB of /tmp/joern-cli-*
was three copies of the Joern CLI I made by hand during today's container
experiments, and 63MB of workspace/ in the repo came from running joern from
the repo root instead of a scratch directory. The product already runs joern
with cwd set to the scratch tree, so its workspace is contained and now
disposed with it. Both are cleaned up; neither was the code's fault, and
conflating my litter with its leak would have hidden the real one.
Disk went from 6.1GB free to 19GB.
857 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`joern` and `joern-parse` are shell scripts that launch a JVM as a separate
process. `subprocess.run(timeout=)` kills the script and leaves the JVM
resident, holding its heap, indefinitely. Every `query_failed: timeout` leaked
about a gigabyte.
Found by running out of memory. After a day of 300s and 2,400s taint timeouts,
four orphaned JVMs -- three hours old, ~2GB between them -- were still resident
on a 7GB machine, and background tasks started being killed. On a CI runner
doing several scans in sequence this is an OOM rather than a degradation, and
it would present as "the scan needs more memory", which is the wrong lesson
entirely: the scan's own peak is a flat ~1.5GB whether its query runs for five
minutes or forty.
The command now starts in its own session, and a timeout SIGTERMs the process
group and SIGKILLs what survives. An injected test runner is left alone,
because those spawn nothing.
Tested with a real shell script that backgrounds a child, since the bug is
specifically about grandchildren -- and the test reaps that child itself if the
assertion fails, rather than leaking the process it is complaining about.
855 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The rank SCALE is generic: `_ACCESS_RANK` maps abstract access levels to
numbers and knows no framework. The EVIDENCE feeding it does not --
`_wordpress_hook_access` holds the literals `wp_ajax_nopriv_` and `wp_ajax_`,
`_rest_route_access` holds `__return_true`. One framework, written in Python.
The consequence is not "slightly worse elsewhere", it is no ordering at all.
Anything unrecognised falls to review-required = 0.30 and ties break
alphabetically by path. That is where CVE-2023-23488's region sits -- position
~1036 of 4,463, below every REST handler -- so a 500-region budget never
reaches it. On a Laravel or Django codebase EVERY region lands in that tier:
thousands of regions ordered by filename.
Three layers, and the model is only the third:
* facts, not code -- an [[access]] fact declares hook patterns and their
access level, the way [[dispatch]] already replaced hardcoded
add_action/apply_filters today;
* generic features needing no framework knowledge -- modelled sinks in the
file, a modelled source present, call-graph distance to the nearest sink,
shipped, an unsanitized path to any sink. This is what would move
CVE-2023-23488's region on its own, and it should exist BEFORE any model,
or the model's contribution cannot be measured against anything;
* the LLM for the residue the first two cannot separate.
Ranking is a safe place for a model because it cannot manufacture a finding:
it decides what is looked at, never what is reported. The arbiters still
decide, the gates stay byte-identical, and a misrank costs recall -- now
visibly, since regions_truncated reports 500 of 4,463.
Records the constraints that must be settled before spending model budget:
batched over a prefilter (4,463 calls a scan is not a CI tool),
leave-one-repository-out (this project already has the train-on-test scar),
a recorded ranking so baselines diff, and honest degradation with no model.
A model proposing FACTS for an unknown framework is a different, slower loop
needing human approval -- ranking at runtime is not policy, a fact table is.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Root cause of every failed whole-repository PHP scan, bisected from 637 files
to one file to one function to one construct:
<?php
function f($n) {
h("p", function($p) { global $g; return $p . $g; });
}
A `global` declaration inside a CLOSURE makes php2cpg 4.0.623 emit a node with
two AST parents, and Joern then refuses to apply its own dataflow overlay:
java.lang.AssertionError: Iterable was expected to have exactly one
element, but it has 2. Hint: trying to resolve astParent ... (which is
probably a malformed cpg)
Controlled, because four earlier hypotheses were refuted by measurement and I
was not going to add a fifth:
global inside a closure WITH use -> ASSERTS
global inside a closure, NO use -> ASSERTS (so `use` is irrelevant)
use, global OUTSIDE the closure -> ok (so scope is the trigger)
The property that matters: one such file costs the WHOLE GRAPH, not itself.
That is why a 2,500-request taint batch and a 13-request dominance batch failed
identically in ~25s, and why I twice called this a per-request cost problem. It
never was. Nothing was ever served; the graph is rejected at load.
Rare enough to exclude rather than surrender to -- 1 of 637 files in Paid
Memberships Pro, 1 of 357 in WP Statistics, 0 of 169 in MW WP Form, about 0.2%
-- so the construct is detected by reading the source, those files are excluded
from the build, and they are reported as `unparsed` so the coverage section
names them instead of implying they were clean.
Repository-scale PHP goes from blocked to 99.8% scanned with a stated gap.
854 tests pass; gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two permanent guards, and a correction that motivates both.
THE CORRECTION. There is no php2cpg defect. Every PHP build on this machine
ran through a local `php` shim proxying stdio through `docker run`, and
php2cpg depends on the INTERLEAVING of its parser's stderr banners with its
stdout JSON to attribute each document to a file. The proxy scrambles it:
streaming shim drops 72-88 767KB / 969KB / 1.2MB different every run
buffered shim drops 0 5,703 bytes empty, and silent
NATIVE php drops 0 2,872,996 bytes identical x3
The 637-file tree with native php: 0 drops, a 4,329,471-byte CPG, 20.0s,
1.9GB peak -- which is exactly what task 5.10 said before I started
"correcting" it. No intermittent failures, no malformed graph, no dataflow
assertion, no scale limit.
GUARD 1: the interpreter must prove it can read the tree, before anything is
built. Every instrument failure this project has had takes one shape -- the
input was unreadable and the tool reported a plausible ZERO -- and it has cost
four wrong diagnoses in a day:
a php that could not see ~/.cache -> "cannot handle 637 files"
a php whose stdio was proxied -> "drops half its batches"
a container missing a bind mount -> "0 dropped", 5,703-byte graph
None announce themselves: file_exists() is false, the parser writes nothing,
the frontend exits 0, and "no errors" reads exactly like success. Checking the
output cannot separate that from an empty repository. Checking that the input
arrived can, and costs one exec.
GUARD 2: the graph census is taken on EVERY build, not only one the frontend
warned about. A graph holding nothing arrives with no warnings at all --
silence is the symptom that has no symptom -- and the shortfall is now
reported as "the frontend reported no failures but the graph holds N of M
source files".
`cpg_build_failed` carries that detail through to the report instead of being
a bare reason.
Also adds tests/test_cpg_queries_compile.py: the queries are 900 lines of
Scala that nothing compiled. A rowsFor signature change shipped with 850 tests
passing and broke every query at runtime, found by a measurement failing hours
later. Each query is now compiled AND run against a real graph wherever joern
exists, and must return a parseable payload.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Sharding was triggered by `Failed to process` warnings. Those warnings are not
evidence about the finished graph:
a build reporting 2 drops -> 3 files, 139 methods, same size as a clean one
The file it warned about was in the graph the whole time. Acting on the
warning is not merely wasteful, it is destructive -- shards cannot see each
other's flows, so splitting a two-file WordPress pair on a false alarm cost it
CVE-2022-25148 outright. That is how this was found: the slices caught it, at
a scale small enough to check by hand, before the whole-repository run could
bury it.
So the build now takes a census of the graph (`census.sc`) when the frontend
warns, and splits only if the graph is actually short of the files handed to
it. A warning is a symptom to investigate, never a verdict. Conservative in
the only direction that matters: if the census cannot be taken, the caller
does the careful thing and splits.
wp-vuln 3 entailed incl. CVE at :225, NO shard (was: 2, sharded)
wp-fixed 2, that one absent
mwwpform 1 / 0
pmpro one-file 18 incl. CVE at :936
pmpro slice 18 incl. CVE at :936, sharded -- genuinely deficient here
The census costs one JVM, and only on a build that warned; a clean build pays
nothing.
849 tests pass; gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two defects the first whole-repository scan made visible. It ran for 23
minutes over Paid Memberships Pro and decided nothing.
**The batch was 97.5% duplication.** `hookCallbacks` describes the REPOSITORY,
not the region, and I put it in every request when the hook half landed. On a
637-file plugin that is 16,270 characters repeated across 2,500 requests:
taint-requests.json 42.5 MB, of which 41.7 MB is one string over again
Joern parsed that with ujson inside a 2GB heap and the batch died in
ForkJoinPool. The driver reported `query_failed` for all 2,500 regions, which
was the honest answer to a question that never should have been asked that
way. Repository-wide fields are now hoisted to a single `--param`, and only if
they are identical across the whole batch -- a field that genuinely varies per
region stays where it belongs.
**Arbitration spent 737 seconds enumerating candidates nobody would read.**
Candidates exist to be put to the judge, and `scan_region` discards them the
moment there is no judge to put them to -- but the driver enumerated them
first regardless, parsing each region's file once per family. With the budget
at zero model calls that is 2,500 parses whose results are dropped on arrival.
build 264.8s query 401.3s arbitrate 737.3s total 1403s
Nobody should pay for a question nobody will ask.
848 tests pass; gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A dropped file is dropped from the ONLY graph there is, so every question
about it has no answer -- which is how a 637-file plugin became a scan of
nothing. But those files build perfectly well alone: the failure is
per-invocation, not per-file. So a file the main build refuses is excluded
from it, given a second CPG, and both are queried with rows concatenated per
request id.
pmpro two-file slice before: cpg_empty, 0 findings
after: 18 entailed incl. CVE-2023-23488 at :936,
reported with cpg_sharded
What this does NOT buy is a flow that crosses the shard boundary: a source in
the excluded file whose sink is in the rest of the tree is invisible to every
question asked, and no merging of results recovers it. That is why the split
is REPORTED rather than quietly served as if it were whole.
The excluded-file tree is symlinks at the original relative paths. A region
asks about `classes/class.memberorder.php` and the queries match a filename by
suffix, so a flattened copy would cheerfully answer about a file nobody asked
about.
Reports gain "What could not be analysed", rendering cpg_sharded, cpg_empty,
files_unparsed, query_failed and budget_exhausted in the reader's terms. A
degradation is a fact about coverage, not a warning to file away: silence in
the coverage table means "no fact table covered this", and silence about an
empty or split graph means something else entirely.
Also removes the WordPress vocabulary from the query, which was the other half
of this request. `apply_filters` was a hardcoded Set in taint.sc and
`add_action|add_filter|add_shortcode` a hardcoded alternation in mapping.py.
Both are now a `[[dispatch]]` fact:
[[dispatch]]
id = "wordpress_hooks"
register = ["add_action", "add_filter", "add_shortcode"]
apply = ["apply_filters", ..., "do_action", ...]
The shape is general -- a registry keyed by a string literal at both ends, as
Django signals, jQuery events and Symfony's dispatcher all are -- so a second
framework is a row in a TOML rather than an edit to a query. `dispatch_apply`
is empty for every language that declares none, and the hook half then does
nothing at all rather than looking for names that mean nothing there.
848 tests pass; gates byte-identical. wpstatistics 3/2 with the CVE at :225 on
the vulnerable side only, mwwpform 1/0, pmpro 18/1, VAmPI 4 unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The intermittent empty-graph builds that cost four manual retries to measure
WP Statistics at all.
First, a correction: the explanation this repo carried was wrong. It said
php-parse's stderr banners interleave into block-buffered stdout and corrupt
the JSON. Capturing the exact bytes php2cpg receives -- by putting a logging
wrapper on `php` -- shows its input arriving complete and valid: two
well-formed JSON documents, 5,176,594 bytes, byte-identical across runs, with
no banner anywhere inside them. The defect is in php2cpg's own reading of that
input, and nothing outside it repairs a given attempt.
Three fixes were tried against the old theory and each changed nothing, which
is how the theory died:
* an unbuffered PHP shim via --php-parser-bin 0 of 6 builds usable
* compacting the JSON, 5,176,594 -> 943,028 bytes 0 of 6
* -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 0 of 6
What is true is that the failure is INTERMITTENT, and that `joern-parse` is
both worse at it and silent about it. Measured on one WordPress slice:
joern-parse 5 of 8 builds usable, and says nothing when they are not
php2cpg 11 of 12 builds usable, and names every file it dropped
So php builds through the frontend directly and retries while it reports
dropped files. Retrying is sound only because the failure is intermittent AND
reported -- retrying a silent failure would be superstition; retrying a
reported one is just using the report. It gives up after four attempts and
reports what is still missing rather than pretending a partial graph is whole.
wp-vuln through the driver before: 4+ manual retries to get one build
after: 6 of 6, 3 entailed findings each time
joern-parse remains the path for every other language, and the fallback when
no frontend is on PATH.
843 tests pass; gates byte-identical; mwwpform, pmpro and VAmPI unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`reachableByFlows` on an empty source list is not a no-op. Joern answers it by
logging
[INFO ] Attempting to determine flows from empty list of sources.
to STDOUT -- while the script is running, so the line lands between the fence
markers, ahead of the payload. `json.loads` then fails on the whole document
and the driver reports `query_failed` for every request in the batch. One
method with nothing to trace from silently cost the answers for all 205 of
the others.
Fixed at both ends, because either alone leaves the seam brittle:
* taint.sc skips the flow query when there is no source to trace from, which
is also the correct thing to do with the question.
* `extract_payload` starts a decode at each line that opens a document and
keeps the first that parses. Trimming to the outermost brackets is the
obvious approach and the wrong one -- `[INFO ]` brings its own bracket.
Found while testing whether `$this` belongs in parameter sources. It does not
belong there on the argument -- the receiver is the object, not request data --
and excluding it cuts Paid Memberships Pro's order class from 16 entailed
findings to 6, all sourced from a named parameter, and makes the 2.9.7/2.9.8
pair separate cleanly.
It also loses CVE-2023-6559: MW WP Form's `_delete_files()` takes no
parameters at all, and the attacker-controlled path arrives as
`$this->attachments`, assigned by another method. So the trade is a VERIFIED
CVE against findings merely suspected of being false, and it is refused. The
receiver stays, with the measurement written next to it. The real defect is
field-sensitivity -- a field should be tainted where it is assigned -- and
deleting the receiver removes the symptom by removing the evidence.
841 tests pass; gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Scanning a two-file WordPress slice returned 64 regions examined, zero
findings and zero degradations. The graph it examined had no methods in it.
php2cpg 4.0.623 feeds several files to one `php-parse` process and reads its
stdout and stderr merged. php-parse writes a `====> File <next>:` banner to
stderr the moment it starts the next file, while the previous file's JSON --
megabytes of it -- is still draining from a block-buffered stdout. The banner
lands inside the JSON, ujson reports
expected json value or } got "=" at index 5011645
and the frontend drops that file. Reproducible on real plugin code, and the
size that matters is one file's AST dump, not the repository's:
class.memberorder.php alone (1834 lines) -> 142KB CPG, no failures
the same file + a two-line file -> 5.7KB CPG, BOTH dropped
One oversized file empties the whole graph. `joern-parse` then does not
propagate any of the frontend's warnings: 13 lines of output, exit status 0,
"Successfully wrote graph". This is what made a 637-file plugin unscannable
and reported it as clean -- it was never a scale limit.
Two fixes, because the failure arrives on two different paths:
* `_unparsed_files` reads the per-file warnings the frontend does emit, and
the scan reports them as `files_unparsed`. This covers the direct-frontend
path where the warnings are visible.
* The batched query now carries a `__census__` of the graph -- method and
file counts -- under a key no request id can collide with, and a scan over
a graph with no methods degrades as `cpg_empty`. This covers joern-parse,
which shows nothing. It rides the batch rather than costing its own
invocation, because avoiding a JVM start is the entire reason batching
exists.
Neither invents a verdict. There is no rung for "the graph was empty" because
the ladder labels verdicts and none was produced, which is exactly why it has
to be a degradation.
pmpro two-file slice before: 0 findings, degradations=[]
after: 0 findings, degradations=[cpg_empty]
Gates byte-identical; 840 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.7, and it was the wrong question.
**The 117k-line pmpro CPG does not load.** Importing it throws in a
ForkJoinParallelCpgPassWithAccumulator during overlay application, so every
query against it returns nothing. It is not a call-graph question at all.
Isolated by running the ARBITER against two prebuilt CPGs rather than
writing a sixth ad-hoc Scala probe, four of which had already failed on my
own mistakes:
small wpdb CPG get_order_by_code -> 1 model_entailed
pmpro CPG getMemberOrderByCode -> 0
Both have their source and sink in ONE function, both were asked with
parameter sources and depth 3, same arbiter and same spec. The CPG is the
only difference.
That clears the facts, which is worth stating: `$wpdb` sinks work, the
parameter-source rule works, and a PHP parameter reaching `$wpdb->get_var`
through a concat is entailed. Whether php2cpg resolves `new X()` and
`$this->method()` is still unmeasured -- it cannot be measured on a CPG
that will not load -- and is recorded as 5.10 along with finding the size
boundary.
**And the defect underneath it, which is the fourth quiet failure found on
this one checkout.** `query_batch` returned `{}` for a failure and `{}` for
a genuine empty answer, so a caller could not tell "could not ask" from
"nothing found". Every query on this scan failed at load and it reported
500 regions examined, no findings, and no degradation: a vulnerable plugin,
clean.
A failed batch now returns None, unparseable output counts as a failure
rather than an empty answer, and the driver records a `query_failed`
degradation naming the kind and the request count.
This is the failure mode the rung ladder structurally cannot catch. Every
other honesty mechanism here labels a VERDICT -- suspicion,
regions_unjudged, regions_unasked, stated family limits -- and when no
verdict is produced there is nothing to label.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Scanning a 637-file WordPress plugin reported 500 regions examined, zero
findings, and a taint query time of **0.05 seconds** -- less than a JVM
takes to start. Dominance ran for 29.56s and config for 22.44s in the same
scan, so only taint had failed.
Linux caps a SINGLE argument at MAX_ARG_STRLEN (128KB) whatever ARG_MAX
says. Measured on that plugin:
taint 2500 requests 1.49 MB <- over the cap
dominance 13 requests 0.04 MB
config 13 requests 0.00 MB
As `--param requests=<json>` execve returned E2BIG, `_run` caught the
OSError, `query_batch` returned {} -- and {} reads as "no rows found". Only
taint crossed the cap, which is exactly why the failure looked like a
result rather than an error.
This is the quiet failure the rung ladder cannot catch. Every other honesty
mechanism in this codebase labels a verdict; here no verdict was produced
to label, and a scan of a vulnerable plugin reported it clean.
The payload is now staged to a file beside the CPG and the path is passed
instead; all three queries read `requestsFile` when it is set and keep
`requests` for the single-request forms and the tests. A test asserts no
argument approaches the cap, since the failure mode is invisible at the
scale the unit tests run at -- batching worked on every small tree it was
ever tried on.
Verified on vampi first: 4 entailed, unchanged, all three known
vulnerabilities still found.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Scanning a 637-file WordPress plugin reported 1577 regions with 0 scanned
and `cpg_build_failed`. The build was not slow or out of memory -- it
failed in 42 seconds -- and `php2cpg` run directly on the same tree
produced a 4.3MB CPG with no errors at all.
The difference is that `joern-parse` detects the language and then shells
out to the frontend in a separate process. On this tree it got as far as
applying overlays and then threw, while still writing an unusable cpg.bin.
Joern's own output recommends the direct route for a large codebase:
"Invoking CPG generator in a separate process... If you are importing a
large codebase (and/or running into memory issues), please try:
invoke the frontend: php2cpg -J-Xmx3072m /tmp/src -o cpg.bin"
So a failed `joern-parse` now retries through `<language>2cpg`, with the
driver passing the dominant region language. A language with no mapping
fails closed exactly as before -- no guessing at a frontend name.
This is the documented fallback rather than a workaround, and it was
diagnosed rather than assumed: the standalone build was run first, and only
after it succeeded on the identical tree was the retry added.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.
The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.
**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.
**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.
**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.
**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.
Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.
Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
JVM startup dominated a scan: a call per region per family put a ten-line
file at 275s over three regions, and would have put a thousand regions at
roughly fifty hours. The queries themselves are milliseconds once the CPG
is loaded, so the cost was almost entirely process starts.
scan_repository() is now collect -> batch -> arbitrate. The three .sc
scripts accept a `requests` JSON map and answer all of them in one run,
keeping the single-request form so the committed measurements stay
reproducible byte-for-byte. Arbiters are untouched: each is handed a
CpgResult whose `run` returns its own prefetched rows.
The batcher builds parameters by calling request_params() extracted from
taint.py, dominance.py and config_value.py rather than reconstructing
them. A second copy that drifts from the arbiter's is the exact shape of
the five family-assumption bugs the predecessor spec spent a session
finding; there is now one definition per family and the batcher borrows
it.
Regions are also deduplicated by (path, function), collapsing the
nameless entry-point rows that were producing `model:injection:?:10`.
Measured on the same ten-line file: 275s -> 106s, three duplicate
findings -> one `model:injection:ping:10`, rung unchanged at
model_entailed. Gates byte-identical.
Server mode is recorded as task 2.3, deliberately sequenced after the
repository measurement: whether the remaining ~100s floor is worth a
process model, a weakened subprocess seam and a Scala-evaluating HTTP
endpoint depends on a build-time-to-query-time ratio only 2.2 produces.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Both were found by running the real engine, and neither was visible offline.
1. A sink was its own sanitizer. The shipped facts mark some sanitizers with a
*shape* qualifier -- parameterized ("execute(sql, params) binds rather than
interpolates") and literal_format_arg ("printf with a constant format string
is safe"). These describe a safe FORM OF THE SINK CALL, not a call that
cleans a value on the path. Flattening them into the sanitizer list made
`execute` both the injection sink and its own sanitizer, so every SQL flow
would have reported as already-clean and nothing could ever be entailed.
Fixed as a class, not an instance: any fact carrying a shape qualifier is
excluded from `sanitizers` and kept in `safe_shape_sinks` for the shape test
task 4.2 will model properly. Fixing only the Python `parameterized` case
would have left C's entire printf family -- which the generalised test then
caught immediately.
2. labeled_family lost its deterministic derivation in group 1. The deleted
classifier resolved a pair's family by declared label, else obligation, else
mechanism, else CWE, and only asked an LLM when every link failed -- which
for a labeled corpus was never. Group 1 reduced this to the declared label
alone, which silently emptied every family whose pairs are labeled by CWE:
the injection slice went to zero pairs. The chain is restored, still with no
model call, and the family census again matches the committed entailment
ceiling exactly (injection 50, memory 176, access_control 8, ...).
Also: the Joern seam now runs in the CPG's own scratch directory, because
`joern --script` writes a `workspace/` beside the working directory and would
otherwise leave one inside the repository being analysed.
The taint query itself was corrected against the live engine: the sink is the
call's ARGUMENTS (data flows into arguments, not into the call node) and
reachableByFlows is called on the sink with the source. With that, the
three-hop chain our flat IR could not resolve is found:
request.args["cmd"] -> cmd -> "echo " + cmd -> full -> os.system(full)
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The whole engine integration is one subprocess boundary: joern-parse builds a
cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between
fence markers, because Joern's stdout carries a banner and a REPL prompt around
whatever a script emits. Nothing is imported in-process, so the core install
keeps dependencies = [] -- a test walks the seam's AST to assert it imports
only the standard library and names no JVM binding.
Fail closed everywhere. A build or query that errors, times out, or prints
something unparseable yields no verdict and the caller reports suspicion. The
taint verdict distinguishes this from absence deliberately: a query returning
None means "the engine could not decide", which is not "there is no flow", and
conflating them is how a missing engine silently becomes a clean bill of health.
The verdict is where the ladder earns its keep. An unsanitized source->sink
path is model_entailed -- the graph establishes it and the LLM's claim is no
longer load-bearing. A path a sanitizer may break is model_corroborated: the
flow is real, but whether the sanitizer suffices is the semantic residual the
LLM answers, checked against this verdict rather than averaged over runs. The
chosen flow is picked by a total order (shortest, then lexicographic), so two
runs over one CPG cannot disagree -- Req 6.4 asserted directly.
The query itself is the part our flat IR could not do: reachableByFlows walks
the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is
exactly the one-hop limit that held measured entailment at 2.2%.
Capability detection reuses semantic/engines.joern_available rather than
growing a second notion of "installed". The audit manifest is regenerated; the
new modules are named by Req 4/5/6 rather than counted as orphans.
Task 2.3, the go/no-go measurement, needs Joern installed and has not run.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
