php2cpg drives PHP-Parser through `php` from a script inside its own install. A second install the
containerised interpreter had no mount for parsed 1 of 637 files and exited 0 -- caught by the build
census, and now refused by name before any build. On 4.0.625 with the closure exclusion gated off:
pmpro 637 of 637 in 100 s, WP Statistics 115 s with only the two data tables excluded. Phase-2 v5
verdict and the within-tier reading recorded in the brief.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
v4 (provenance split): pmpro 69 and wpstatistics 172 unchanged -- their tier-4 members are source-fed
through same-file helpers that read the request; mwwpform 68 -> 248, because its CVE's region is fed by
a constructor parameter, the shape the split demotes and the exact flow had put at 30. v5 asks the flow
for exactly the parameter-fed fields a sink-bearing region reads, memoised per (family, file, field),
and reports carriedBy for diagnosis.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Structural carried (v3) measured: pmpro 391 -> 69, wpstatistics 504 -> 172, mwwpform 513 -> 68 -- and
tier 4 swollen to 555 and 569 regions, because "assigned from a parameter" was folded in with "assigned
from a source" and every constructor does the former. The query now reports carriedKind; the vector
promotes on "source" and gives "parameter" a within-tier weight of 0.75.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A pass over every region of pmpro (22,315 pairs) took 255 s without the join facts and past the scan's
300 s default with them, and a timed-out pass answers nothing -- reported once as tiers={} and the static
order. mwwpform with carried: 292 -> 30.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
budget_at_recall by a real evidence pass over every region: pmpro 391 -> 116, wpstatistics 504 -> 223,
mwwpform 513 -> 292, vampi 16 -> 16. The gate (< 50) is not met. pmpro's CVE is inside its 121 tier-4
regions; WP Statistics's and MW WP Form's are one tier low because `carried` -- stage one of the
two-stage join -- was not reported as evidence. taint.sc now reports it where a sink exists, and the
vector reads the query's answer when given.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
WP Statistics's first region is JavaScript, and a build asked for as JavaScript went looking for
jssrc2cpg, failed both launchers, lost the whole run's output and left a 9.6 MB scratch directory behind.
Also notes upstream: joern#6281 is fixed in v4.0.625.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`Evidence.score` orders pairs within a tier by one table of weights (open sinks, cleansed sinks, local /
carried / near sources, entry, declared public), capped so volume stops counting, and zero for tier 0 so
the weights can rank but never manufacture. `ScanBudget.order_by_evidence` runs the evidence pass over
every region and cuts `max_regions` from each region's best (tier, score); `order_by_evidence()` is the
one ordering function, used by the scan and by `benchmarks/ranking/position.py --evidence`. Off by
default until the position benchmark says where the pinned CVEs land under it.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Ten real pmpro pairs at callDepth 3 returned 3,838 flow rows. Every field the driver reads is a property
of the sink or of (source kind, source, sanitized); only `length` varies per path, and the driver ranks
by the shortest. Grouped on that key with the minimum length and a `paths` count: 1,155 rows carrying
3,838 paths, same time. The specs are corrected too -- the first write-up called the depth-3 rows
"mostly the same evidence repeated", and 3.3x is what the measurement says; the other 30x is distinct
sources.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
pmpro, 637 files, the product's own build path after 60963f5: build 97 s with overlays, evidence 76 s,
taint 4,105 s for 348 real requests (11.8 s each), arbitrate 99 s, peak child RSS 2.4 GB. The CVE is at
class.memberorder.php:936 among 250 entailed findings across 40 files -- the number the flow-aware-ranking
spec now has to concentrate. callDepth measured on the same graph: a 1.4x time term and a 100x payload
term (3,838 rows for ten requests at depth 3 against 38 at depth 1).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.13, measured on ten real pmpro pairs at callDepth 1 with each join half switched separately and
the 54 s JVM fixed cost subtracted: both halves off 11.1 s/request, field half 10.9 s, hook half past
49 s (timeout). Stage timers put it in `hookSeedsIn("")`: every field access in all 7,061 methods put
through the field join before the first callback was answered. Seeds are now the callback's own body
and file, which is where WP Statistics's `set_current_page` and `$this->rest_hits = ...` both live.
Both halves on: 9.7 s/request. `sourceNodes` is also materialised once per request instead of once
per sink.
The WP Statistics re-verification then exposed the larger term. A `joern --script` run is two JVMs and
`-J-Xmx` sizes only the launcher, so the heap knob governed nothing; the backend now sets
JAVA_TOOL_OPTIONS too. And since PHP builds went through php2cpg directly (1ce7e9d, an hour after the
hook half was verified from a joern-parse build), every query batch has recomputed the dataflow overlay
at load -- 54 s on pmpro, and on WP Statistics an OutOfMemoryError inside OssDataFlow after 21 minutes
at 2 GB, no completion in 25 minutes at 4 GB, and no help from --max-num-def at 1000 or 200. Two
vendored browser-profile tables, one array literal each, were 60% of that graph.
Three changes, all named in the log: `queries/overlay.sc` applies the overlays once at build time and
the saved graph replaces the raw one (joern-parse --overlaysonly NPEs in 4.0.623); a source file over
1 MB is excluded as a data table and reported in `unparsed`; and the seed scoping above. After: the WP
Statistics graph reloads in 8 s, the `record` request that timed out at 520 s answers in 39 s including
the JVM, and CVE-2022-25148 is entailed again with `sourceKind = hook`. A raw-vs-saved payload diff on
pmpro shows identical evidence keys and fewer duplicate flow rows.
Also: the backend's own unit tests left fifteen ousast-cpg-* directories in /tmp per run; conftest now
keeps tempfile under the test's directory. The compile test covers overlay.sc.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A pair with no sink of its family in reach cannot yield a flow, in any
language under any dataflow model, so its taint request is never issued. The
count is recorded as requests_pruned -- exact, so not a degradation -- and a
test asserts that every other pair's request still goes out.
637-file plugin, 500-region budget
tier_counts {0: 2161, 1: 60, 2: 20, 3: 211, 4: 48}
pruned 2,161 of 2,500 = 86% evidence pass 94.5s
taint 339 requests -> 2,400s TIMEOUT
Gate met and exceeded; the scan still does not finish, and the reason corrects
phase 0. "1,148 ms per request" was an average over a 250-request set that was
88% tier 0 and answered instantly; the ~30 real requests in it cost ~9.5s each.
Pruning removes exactly the cheap questions and keeps the expensive ones, so it
cannot make them faster. The honest cost of a taint question that CAN have an
answer is 7-10s at callDepth=3, and that is what phase 2 and 5.13 design
against -- not 1.1s, and not 6.6s either. Both specs corrected.
869 tests pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Read against today's measurements the brief had two errors and one gap. They
are corrected in place and kept on the record, because the corrections are
themselves results.
Phase 1's gate was wrong. Pruning removes 53% of requests, and 1,177 x 6.63s is
still 2.2 hours -- it halves the problem and does not finish the scan. Saying
otherwise would send someone looking for a bug in a phase that did exactly what
it can.
Ranking's job is concentration, not inclusion. CVE-2023-23488's region is at
position 390, inside the budget, examined. It goes unreported because 500
regions are unaffordable at 6.63s per family per region, not because it was
left out. The metric is therefore budget_at_recall -- the smallest K holding
every pinned CVE -- and phase 2's gate is that K below 50.
The gap: the heuristics were named, not defined, and an undefined heuristic
cannot be checked for language leakage. Now defined as an evidence vector over
(region, family) pairs whose every field is stated in terms of fact-table
categories -- source, sink, sanitizer, dispatch, access -- so a heuristic that
mentions `$wpdb` has by construction left the vector.
The architectural decision that makes it agnostic rather than merely
neutral-in-intent: the vector is a CPG query WITHOUT dataflow, a sibling of
taint.sc that never calls reachableByFlows. Tiering is the CPG query without
dataflow; arbitration is the query with it. Same graph, same facts, two costs.
The graph normalises syntax across frontends, so "this argument is a literal"
means the same thing in PHP, Python and C.
Five tiers as predicates over the vector. Only tier 0 excludes, and its
predicate has a proof: a family with no sink in reach cannot produce a flow in
any language under any dataflow model. Its scope must match the arbiter's
callDepth or it is not exact and must not be called tier 0. Everything above
orders, where being wrong costs position, which regions_truncated reports.
Where the model helps is narrower than "ranking": ambiguous cleansing (esc_sql
two statements earlier into a variable), unfamiliar sink shapes as an ordering
opinion never a fact, and within-tier ties. Where it must not be allowed:
computing tier 0, seeing the question "is this vulnerable", or seeing the
arbiter's own verdict on the same run.
Six metrics so every phase can fail, and budget_at_recall is the one that
decides the release question: below 50 and the model is a refinement, above
and the model is the plan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A taint request costs 6.63s, a 637-file plugin yields 4,463 regions, and 83.8%
of them share one rank ordered alphabetically. The ranker is not weak, it is
looking at the wrong thing.
The evidence that this is tractable, from one plugin:
getMemberOrderByCode (holds CVE-2023-23488) rank 0.30
$wpdb->get_var("... WHERE code = '" . $code . "' LIMIT 1")
getMemberOrderByID (sibling) rank 0.30
$wpdb->get_row("... WHERE id = '$id' LIMIT 1")
pmpro_rest_api_get_membership_level rank 0.80
no SQL sink at all
A region with no sink outranks two holding raw concatenated SQL. The signal is
in the sink call's own text -- interpolated versus bound -- and nothing in the
ranker reads it.
And the detail that decides whether a keyword rule suffices: `esc_sql` IS in
scope for the CVE's function, applied to a different statement. "A sanitizer is
present, rank it lower" demotes exactly the right answer. Knowing which call it
wraps is reading, not matching -- which is the case for a model rather than a
table.
Unit: (region, family) PAIRS, tiered by cheap evidence. Tier 0 (no sink of that
family in scope) is the only tier that excludes rather than orders, so it alone
must be exact; above it, being wrong costs position rather than the finding.
That bound is what makes a model safe here -- a ranker cannot manufacture a
finding.
Six phases, each shipping only if it beats the previous one on a repository it
was not developed against: baseline, exact pruning (the phase that makes a scan
finish), heuristic tiers, the LLM ranker, breaking the closed loop, then prompt
optimisation. Phase 2 carries an explicit possibility that the model is not
needed for tiering -- and knowing that is worth more than assuming it. Phase 5
comes last because an optimiser run against a censored metric optimises the
censorship.
Records what the prompt must carry, and that the question is never "is this
vulnerable" -- that is the arbiter's job and the one thing a ranker must not be
asked.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Feeds the day's measurements back into the spec and reorders the roadmap
around what is actually blocking.
STATUS. Three CVEs found on real WordPress plugins -- CVE-2023-23488,
CVE-2023-6559, CVE-2022-25148 -- each entailed by the graph with no model in
the loop and each absent from its fixed side. Two of them require the
two-stage join and would be invisible without it. The detector is not the
problem.
The repository is. A 637-file plugin builds in 76s to a 4.25MB graph,
dominance and config answer, taint does not finish. Three blockers in the
order they bite: cost (6.63 s/request), ranking (84% of regions share one rank,
ordered alphabetically), coverage (access_control never asked for PHP,
output_encoding untrustworthy while the sanitizer list is flat).
And what is NOT a blocker, contrary to most of what this file said this
morning: php2cpg. One genuine defect, 0.2% of files, detected and excluded and
reported. Everything else attributed to the frontend was a local php shim
proxying stdio, which cost four wrong diagnoses in a day.
NEW TASK 5.14, and it is the cheapest thing on the board. 53% of taint
requests are put to a family whose sinks are not in scope -- all 487 of PMPro's
deserialization requests go to a repository that never calls unserialize.
Skipping them cannot lose a finding, because a family with no sink cannot
produce a flow. 4.6 hours -> 2.2, before any ranking work.
ROADMAP reordered around one question: the shortest path to a scan of a real
plugin that reports its CVE. Prune the impossible questions, then rank (an LLM
job, and safe as one because ranking cannot manufacture a finding, with the
closed-loop problem treated as a design requirement rather than a caveat), then
attack the per-request cost only if those two are not enough.
857 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
DeepSeek announced V4.1 Flash on 2026-09-10 and routes V4 Pro requests to it
from 12:00 Beijing time on 2026-09-14, billed at Flash's price.
The consequence worth writing down is not the id change. It is that the
detector and the judge are now the SAME model, where the design assumed the
judge was a stronger and more expensive second opinion. Every claim in this
repo that a judge call is a higher-quality check is now a claim about the
PROMPT and the evidence put to it, not about model class -- and the residual
question a judge is asked ("does this sanitizer suffice for this weakness")
has to carry its own weight.
Legacy ids stay in the price table so a recorded run from before the migration
still costs out; a manifest that cannot price its own model is one nobody can
audit.
The code for this landed in 83ab533, swept in by `git add -A` under a docs
message about the ranker -- the second time today. This commit is the record
it should have had.
855 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three corrections and one design requirement, from measurement and from
maintainer direction.
MEASURED. 4,463 PMPro regions carry five distinct ranks, and 3,742 of them --
83.8% -- share rank 0.30, ordered alphabetically by path. That tier spans
positions 159..3,900, and 341 of the 500 examined regions come from it: two
thirds of the budget spent on an arbitrary alphabetical slice. It is not a
weak ranker, it is barely a ranker.
CORRECTED. CVE-2023-23488's region is at position 390 of 4,463 -- INSIDE the
budget and examined. I stated repeatedly that it ranked ~1036 and was never
reached; that number came from before the array-callable and method-recogniser
fixes moved it. The reason it goes unreported is the taint query timing out,
so no region got a verdict at all. Ranking is a real problem; it is not this
miss, and conflating them would have sent the fix to the wrong place.
DIRECTION. Ranking is the LLM's job, not a tie-breaker for a residue. The
judgement "is this worth looking at" is a judgement about unfamiliar code.
Facts and generic features stay as cheap evidence for the model to read and as
the fallback when none is configured.
AND THE LOOP IS CLOSED, which is worse than train-on-test and was the sharper
point. A region the ranker excludes is never arbitrated, so it never produces
a label, so the ranker never learns it was wrong -- the feedback is censored by
the policy being trained, and every run confirms the ordering it already had.
Optimising on that makes it more confident, not more correct.
Breaking the loop is therefore a design requirement, with two cheap mechanisms
recorded: an exploration slice (~10% of budget sampled from outside the top-K,
the only labels the policy did not choose) and full labelling on repositories
small enough to arbitrate every region, which is policy-independent ground
truth. Inverse-propensity weighting is noted as available if the ranking is
made stochastic, and as unnecessary otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Four configurations over one verified graph (637 files, 7,061 methods, census
confirmed BEFORE anything was timed) and one 250-request set:
both joins off 1657.5s 6,630 ms/request 5779 rows
field half only 1623.0s 6,492 ms/request 5779 rows
hook half only 1781.0s 7,124 ms/request 5779 rows
both on 2026.3s 8,105 ms/request 5779 rows
The base query is the cost. The field half is free -- inside noise, measured
slightly faster -- the hook half adds 7.5%, both add 22%. Cutting them buys a
fifth of the time and costs two of the three CVEs.
Corrects a number I published earlier: ">0.94s per request" assumed taint
completed within its 2,400s ceiling. It timed out, so that was a floor dressed
as an estimate. The real figure is seven times higher.
2,500 requests is ~4.6 hours; all 4,463 regions would be ~41. Repository-scale
PHP is not slow, it is the wrong shape of question asked thousands of times.
Caveat kept with the numbers: all four configurations return an identical 5,779
rows because these are the TOP 50 regions -- REST handlers where framework
sources already exist and the joins add nothing. The joins earn their keep
lower down (`_delete_files` at rank 0.30). This measures their cost faithfully
and says nothing about their value; the two must not be conflated when deciding
what to cut.
Levers, none of them the arbiter: fewer and better regions (5.12, ~10x on its
own), fewer families per region (five are asked of every region regardless of
whether its scope holds a sink of that family), and callDepth -- three levels
over 7,061 methods is what makes a request cost seconds.
Also notes that commit 7308570 carried the OPENULTRASAST_CPG_QUERY_TIMEOUT
change under a docs message; `git add -A` swept it in and the message does not
say so.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The rank SCALE is generic: `_ACCESS_RANK` maps abstract access levels to
numbers and knows no framework. The EVIDENCE feeding it does not --
`_wordpress_hook_access` holds the literals `wp_ajax_nopriv_` and `wp_ajax_`,
`_rest_route_access` holds `__return_true`. One framework, written in Python.
The consequence is not "slightly worse elsewhere", it is no ordering at all.
Anything unrecognised falls to review-required = 0.30 and ties break
alphabetically by path. That is where CVE-2023-23488's region sits -- position
~1036 of 4,463, below every REST handler -- so a 500-region budget never
reaches it. On a Laravel or Django codebase EVERY region lands in that tier:
thousands of regions ordered by filename.
Three layers, and the model is only the third:
* facts, not code -- an [[access]] fact declares hook patterns and their
access level, the way [[dispatch]] already replaced hardcoded
add_action/apply_filters today;
* generic features needing no framework knowledge -- modelled sinks in the
file, a modelled source present, call-graph distance to the nearest sink,
shipped, an unsanitized path to any sink. This is what would move
CVE-2023-23488's region on its own, and it should exist BEFORE any model,
or the model's contribution cannot be measured against anything;
* the LLM for the residue the first two cannot separate.
Ranking is a safe place for a model because it cannot manufacture a finding:
it decides what is looked at, never what is reported. The arbiters still
decide, the gates stay byte-identical, and a misrank costs recall -- now
visibly, since regions_truncated reports 500 of 4,463.
Records the constraints that must be settled before spending model budget:
batched over a prefilter (4,463 calls a scan is not a CI tool),
leave-one-repository-out (this project already has the train-on-test scar),
a recorded ranking so baselines diff, and honest degradation with no model.
A model proposing FACTS for an unknown framework is a different, slower loop
needing human approval -- ranking at runtime is not policy, a fact table is.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two permanent guards, and a correction that motivates both.
THE CORRECTION. There is no php2cpg defect. Every PHP build on this machine
ran through a local `php` shim proxying stdio through `docker run`, and
php2cpg depends on the INTERLEAVING of its parser's stderr banners with its
stdout JSON to attribute each document to a file. The proxy scrambles it:
streaming shim drops 72-88 767KB / 969KB / 1.2MB different every run
buffered shim drops 0 5,703 bytes empty, and silent
NATIVE php drops 0 2,872,996 bytes identical x3
The 637-file tree with native php: 0 drops, a 4,329,471-byte CPG, 20.0s,
1.9GB peak -- which is exactly what task 5.10 said before I started
"correcting" it. No intermittent failures, no malformed graph, no dataflow
assertion, no scale limit.
GUARD 1: the interpreter must prove it can read the tree, before anything is
built. Every instrument failure this project has had takes one shape -- the
input was unreadable and the tool reported a plausible ZERO -- and it has cost
four wrong diagnoses in a day:
a php that could not see ~/.cache -> "cannot handle 637 files"
a php whose stdio was proxied -> "drops half its batches"
a container missing a bind mount -> "0 dropped", 5,703-byte graph
None announce themselves: file_exists() is false, the parser writes nothing,
the frontend exits 0, and "no errors" reads exactly like success. Checking the
output cannot separate that from an empty repository. Checking that the input
arrived can, and costs one exec.
GUARD 2: the graph census is taken on EVERY build, not only one the frontend
warned about. A graph holding nothing arrives with no warnings at all --
silence is the symptom that has no symptom -- and the shortfall is now
reported as "the frontend reported no failures but the graph holds N of M
source files".
`cpg_build_failed` carries that detail through to the report instead of being
a bare reason.
Also adds tests/test_cpg_queries_compile.py: the queries are 900 lines of
Scala that nothing compiled. A rowsFor signature change shipped with 850 tests
passing and broke every query at runtime, found by a measurement failing hours
later. Each query is now compiled AND run against a real graph wherever joern
exists, and must return a parseable payload.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Four instrument failures in one day, all the same shape: the input was
unreadable and the tool reported a plausible zero. file_exists() false, parser
writes nothing, frontend exits 0, and "0 dropped, 5,703-byte graph" reads
exactly like success.
* a docker php shim not mounting ~/.cache -- twice, hours apart, the second
time after I had already diagnosed the first
* a measurement harness whose script path did not resolve from its cwd,
which printed 0.5s and `rows=?` where a JVM takes 15s just to start
* a container that could not follow the symlinks in the tree it was given
The rule that catches all four is one line and runs before the work: make the
tool open one input and fail loudly if it cannot. Every mechanism in this brief
so far checks the OUTPUT; this checks that the input arrived.
Also records that nothing compiles the CPG queries. A signature change to
rowsFor shipped with 850 tests passing and broke every query at runtime -- found
by a measurement failing, not by CI. The canary already proposed here covers it,
since a query that cannot compile cannot entail the canary.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
