Committed before any case is scanned: what is under test (the analyzer at the
runner's commit, per-case family only, four scans per case), how a finding
MATCHES a case (a fix-changed file, the case family, the named function or
within 10 lines of a fix hunk), detection / fixed-side / benign false alerts,
unanswered and instrument failures, the precision sampling rule (every third
pooled vulnerable-pin finding, or all if 15 or fewer), what is reported
against the M4 gates, and that no retune may be informed by the results.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every reserved case verified with git and file reads only -- nothing
scanned: pins present and ordered, the sink present at the vulnerable pin,
the fix touching the sink (or LearnPress's declared source-side fix site),
and the sink file hashed at both pins (benchmarks/independent/freeze.py,
record freeze-v1.json).
Each case now carries a benign control from its own history, reviewed by
diff. The mechanical keyword filter was not enough: it proposed an access
restriction, a filter check, API-token scopes, a login change and a
TLS-verification option, all rejected; Ultimate Member and alerta got
hand-picked controls (a WPCS reformat; a dependency change with both sides
still vulnerable). The guard test requires a benign pair per case once
frozen.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Eleven cases chosen by the maintainer from fifteen researched by advisory
and fix diff only -- nothing cloned, nothing scanned, none among the 262
repositories the benchmarks reference -- plus the earlier Ghost
reservation. Four are PHP repositories that ship JavaScript (two WordPress
plugins), satisfying the multi-language requirement; families are limited
to the v1 scope (injection, untrusted destination, permissive config).
The development subjects shaped every rule of the last week and can no
longer measure whether those rules generalise; this population can, until
something scans it. A guard test fails the moment a reserved repository is
referenced by a recipe, catalog, manifest or measurement outside
benchmarks/independent/. Freezing (pins and sinks verified by reading
source, a benign change per case) is the next step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Asked of every parameter up front, 6ebbc89's caller check ran a repository-
wide flow query per parameter of every method in a file and cut PMPro's
completed injection questions from 272 to 67 in the same budget -- so its
48 -> 26 findings could not be read as precision. The flow into the field
is asked once as before; only a parameter at the head of an unsanitized
flow is checked against its callers. finding_dump now records completed
regions so runs compare only where both answered.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
48 findings, every 5th reviewed against source: 2 true (both privileged),
8 false; 6 of the 8 share one cause, object fields treated as attacker
input when they are loaded from the database. benchmarks/push/finding_dump.py
writes every finding of chosen families for review.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The loop the maintainer called out: four scale defects in the PHP transfer
replay, found one per two-hour Joern run, all Python bookkeeping between
engine calls. benchmarks/push/python_overhead.py ends that: the real
replay() on the real PMPro pins, with a JoernBackend whose graphs are stubs
and whose queries answer instantly with rows recorded from the real engine.
Engine time is zero, so everything measured is ours, in about a minute,
under the sampling profiler. Its first pass named two more walls:
- Region discovery re-parsed the ruleset for every region: 89,127 TOML
parses, 275 s of a 339 s discovery, twice per push. The fact, obligation
and taxonomy loaders now keep the parsed (immutable) result keyed by each
file's name, size and mtime -- an edit in place or an added file reloads.
Discovery 339 s -> 7 s; output digest identical on PMPro.
- affected_context decoded every span's and every line anchor's path for
every context row -- a changed file has an anchor per unchanged line --
78% of a stubbed push. Decoded once per call, anchors indexed by line and
bisected, original order kept. A randomised control compares it against
the row-by-row matching over 200 contexts (>100 attachments).
Result: the whole stubbed PMPro push, 27,542 answered requests and 2,688
outcomes, runs in 58.5 s of Python. What is left is the engine.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
b4b9308 was committed with two failing tests: the module audit read
`from . import profiling` as an import of the package, called profiling
orphaned, and the committed audit manifest drifted. My check read tail's
exit status, not pytest's. Now a from-import; manifest regenerated (109
load-bearing, 0 orphaned); 1,444 pass with pytest's own exit checked.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The cache fsynced every payload and manifest -- two per entry, ~18 ms, and a
push publishes one entry per question (~22,000 on a WordPress plugin). A
read already rejects any entry whose manifest receipt, payload size or
payload SHA-256 does not check, so a torn write is a miss, never a wrong
answer; the fsyncs bought nothing the verification does not.
The broader problem: three quadratic walls in this replay were found one
per two-hour run, because all were Python bookkeeping between engine calls
and the only record was the engine log. `OUSAST_SAMPLE_PROFILE=<path>`
starts an in-process sampler that rewrites the hottest stacks every 30 s,
so a run killed at its deadline still names where its time went. The host
cannot ptrace the container, hence in-process. The replay harness turns it
on per case.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The 13.3 replay on 7cf256b crashed on case 1: `pre-push` ran past its
7,200 s deadline plus the harness's 60 s grace, `subprocess.run` raised
TimeoutExpired, the harness died without running cases 2 and 3, and the
case's captured output died with it. Each case now streams stdout and
stderr to files with INFO logging on; an overrun kills the case's process
group, is recorded as `deadline_overrun` with its size -- the product
failing its cancellation contract -- and the next case still runs.
`--only` reruns selected cases. CLI log lines now carry a timestamp, so a
log can say where the time went.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The PHP transfer replay (pre-push 13.3) spent its whole 7,200 s case in the
head side's evidence pass. A probe on one PMPro graph named the cause: not
compute -- context adds ~20% engine time -- but OUTPUT. Each evidence request
listed every method its scope reached, up to 1,845 rows; 200 requests into
the bundled Braintree SDK returned 52,831 rows, 9.4 MB, against 211 rows
without context, over 22,125 requests in a 3 GB container.
`affected_context` only ever attaches a location in a changed file, and
every other row was checked (real file under the root, usable extent) and
discarded. The scan now sends the changed files; the engine itemises those
and folds the rest into one `context_elsewhere` row carrying exactly what
that check reads. With no safe file list the engine itemises everything,
as before.
Equivalence: unit controls over valid, missing-file, no-extent and
outside-root cases, with and without a changed-file row; a real-Joern
control; and on PMPro itself, 2,440 relationships and zero gaps decided
identically from both forms, 52,831 rows down to 3,241 with the most
referenced file as the change. Probe: benchmarks/push/context_cost_probe.py.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Same 500-region PHP run on frozen source: completion 281/406 against 153
(hard cap) and 131 (7.8), no portion killed, per-family findings never
lower, declared target still entailed. Records the 7.9 task, its limits,
and corrects the 7.8 reuse note, which was stated backwards.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Final end-to-end on the 500-region PHP subject: findings identical at 125 either way, completion
slightly lower under portioning (131 of 406 vs the hard-cap-plus-reserve 153), wall time higher
(2,982 vs 2,338 s). Told straight: portioning does not beat the cap on throughput here; it matches
it on what it finds, and what it buys is kill-safety -- a killed batch keeps its streamed answers
instead of losing them. Reaching the same 125 findings is the evidence that streaming loses nothing
the cap kept.
Completion did not rise because the requeue re-asks a killed portion remainder, and on this subject
those are the expensive output-encoding questions that keep hitting the ceiling in smaller pieces,
spending budget without completing. Telling a transient kill from a too-expensive family is the next
lever, not more portioning. Neither approach is release-ready anyway: 32-38% completion against a
95% gate on development data that qualifies nothing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The helper took a destination from the caller and rm -rf it. Re-running it into the same path while
a container had that path mounted deleted taint.sc out from under a live scan, and every request
after printed no such cpg query -- the live-mount hazard the helper exists to prevent, in a new
form. It now mints base/frozen-<commit>-<pid>-<n> itself and prints it, so reuse is impossible by
construction and the caller captures the printed path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every long run mounts src into a container, and mounting the working tree is a live mount: edits
made during a multi-hour run are picked up mid-run, and a measurement whose code changed underneath
it is not a measurement of anything. A five-hour PHP transfer replay was lost that way today.
The export has to be checked too. git archive piped into tar once produced a tree missing the whole
ontology -- every tracked file after groovy alphabetically -- and reported success, because tar
status in a pipeline was never read. Every scan on that tree died at its first fact load and looked
like a slow run for fourteen minutes.
So: archive to a file, extract, count against git, and refuse an export without the ontology.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
classes/class.memberorder.php:936:getMemberOrderByCode, injection. A value concatenated into SQL with
no prepare, which is the declared M3 target for PHP, found for the first time.
Nothing about detection changed. The population became affordable: tier-zero pruning scoped to its
partition, batches budgeted by the sink work they imply, a per-query timeout no longer cancelling the
scan, and a reserve kept for arbitration. Before those this population completed zero questions at
every budget tried.
At 500 regions with a 3,600 s deadline: 271 of 406 questions completed, 213 findings, 2,950 s, and
eight findings in the target file including the target itself.
The earlier bounded attempt failed for a reason no budget increase could fix -- 25 regions of 4,461
is 0.6 percent of the population and the target was never asked about -- so this sizes the replay
before spending hours on it. One scan at this budget is 2,950 s, a push transaction is two scans plus
discovery and preparation, so the three declared comparisons are roughly five and a half hours.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
A census that reports what it found cannot say whether a defect it missed was examined and cleared or
never looked at, and that is the difference between a negative and a silence. It is also how a
bounded run budget gets sized without guessing: task 13.3 needs to know at what population the
declared PHP target enters scope, and asking the record is cheaper than binary-searching with
forty-minute scans.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PMPro at 25 regions, which completed before this work and still does: 27 questions completed and 19
findings before and after, injection 1 and output encoding 18 in both. The batching changes what a
large population can afford and nothing about what a small one answers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the 500-region population where every earlier run produced nothing: 153 of 406 questions completed
and 125 findings, inside the deadline. 380 requests carrying 5,807 sink visits split into 11 batches,
each answering in 80 to 197 s, none near the 300 s ceiling. Every family that can speak on this
subject now does, injection included, which is the one holding a true positive here.
Four defects, each hidden by the one before it, and three of them one rule.
Batches sized by request count, when cost is per matched sink.
Packing heaviest-first, which starved injection of all 114 of its questions.
Any per-query timeout cancelling the whole scan, abandoning batches with 33 minutes left.
No budget kept for arbitration, which removing that cancellation exposed: rows are not findings.
The reserve had to be checked per BATCH. Checked between query kinds it changed nothing at all,
because one kind is a single loop iteration that runs every one of its batches inside it.
What it does not claim: 253 of 406 questions still went unanswered and one batch still records a
timeout; the raw finding count is dominated by output encoding, the family already measured to be
imprecise on PHP; and the reserve share of a fifth is a first value, not a tuned one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Open since September and re-measured nine times. The cost of a taint batch is the number of CALL
NODES a family sink names match in the repository. Not the request count, not the declared sink
count, not any request parameter. reachableByFlows is asked per matched sink, and PHP
soutput-encoding sinks are echo and print, which a WordPress plugin writes thousands of times.
Three controlled sweeps on one graph, built once.
Batch size exonerated: 400 requests answer in 170 s, and cost per request FALLS with size because
roughly 70 s is fixed JVM start and graph load.
Request parameters exonerated: at 200 requests the full production shape, call depth 3 with
parameter sources, costs 221 s against a 150 s baseline.
The family is the variable: at the same 100 requests, SQL injection 147 s with six declared sinks,
path 194 s with sixteen, output encoding 704 s with seven.
And the driver named its own expensive step once it was asked to: 2,435 evidence requests in 45.5 s,
then 315 arbitration requests unanswered at 1,200 s, then a split abandoned for want of time.
It survived since September because every reproduction outside the driver carried the SQL sinks --
the family the declared target belongs to, and the one family that is cheap on this subject. Nine
hypotheses were eliminated by re-measuring the cheap case.
Splitting never rescued it because a failed batch is split by REQUEST COUNT and each part carries the
same everywhere-sinks. contributor-scan 7.7 carries the fix: budget a batch by the sink nodes it
implies, using the count the evidence pass already computes, and report a family whose sinks are
everywhere as expensive rather than as failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Four census runs on this subject spent whatever query ceiling they were given: 363 s of 300, 1,255 s
of 1,200, 1,874 s of 1,800, at question counts of 2,526 and 406. Cost that tracks the budget rather
than the work is not per-question cost, and tier-zero pruning cutting the count six-fold changed
nothing. A census runs one batch, so no census could answer whether the batch itself scales.
One graph, built once, five batch sizes, every one answered:
25 requests 71.4 s 2.86 s each
50 requests 76.0 s 1.52 s each
100 requests 155.4 s 1.55 s each
200 requests 166.7 s 0.83 s each
400 requests 169.7 s 0.42 s each
Sublinear, and roughly 70 s of it is fixed JVM start and graph load. Four hundred dataflow requests
answer in 170 seconds on the graph production cannot finish 406 on.
So the difference is in the request PARAMETERS. The probe sent callDepth=0 and no parameter sources;
production sends callDepth=3 with parameter sources on every entry point, plus a hook table, a
dispatch list and context evidence. Task 13.2 recorded in September that every reproduction outside
the push path answered in under a minute and the difference was not known. Size is now removed as a
candidate and a four-variant probe is running to isolate the rest.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The mechanism already existed. A question whose evidence vector says no sink of its family is in
reach cannot produce a flow, and tier-zero pruning defers it. That was switched off for the whole
scan whenever any of twelve degradation reasons appeared, and a multi-language repository always
carries two of them, so the measured plugin arbitrated every question it could ask. Third instance
this week of one blast radius, and the invariant written into the design on Monday named it.
A prune blocker that names a partition now withholds pruning from that partition questions and from
nobody else. cross_partition_semantics_unresolved leaves the blocker set entirely, for the reason it
left the integrity gaps: a PHP family sink is PHP, so a second language cannot be why its vector saw
none.
Measured on Paid Memberships Pro. At 25 regions arbitration falls from 127 questions to 34 with 93
recorded deferred as tier_zero, and 19 findings before and after, identical family by family. At 500
it falls from 2,526 to 406 with 2,120 deferred.
It is not sufficient, and that is the useful half. The 500-region run still fails in the same way at
the same cost, because 406 questions at roughly one and a half seconds each still exceed the
300-second per-batch ceiling. This lever moved the volume; the ceiling is a separate limit.
The census now records what a scan skipped and why, per family, so a cheaper scan and a narrower one
stop looking identical: a family whose every question was pruned reports no sink of this family in
reach rather than region offered, no question asked.
1,420 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat models 18 of 175 distinct call names, covering 12 percent of 1,185 calls. That share is not
a score: most of what a program calls is not security-relevant, and a high number would mean the
tables were too loose. It is a baseline to move deliberately.
The top product candidate is res.render, 23 calls with five in a method that reads untrusted input.
Task 14.3 reasoned its way to that same gap by hand and declined to model it as a flow sink; the
instrument found it in one run, which is the validation that matters. It also surfaced Math.random in
the user data-access module, and weak randomness is a weakness class this ontology has no family for
at all -- so no family census could have found it, because the family list is itself the blind spot.
That is the kind of gap only a coverage instrument finds.
It had to be taught what is not an API. The first run ranked _tmp_3.push and __ecma.Array.factory
first, which is how the frontend lowers a destructuring and an array literal, and a Cypress suite and
a Gruntfile held four of the next five. Both are separated now, because noise at the top of a
worklist is how a worklist stops being read.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Five of seven families established nothing in the family census, and every one failed for an ontology
reason with the flow engine ready and nothing to aim at. Each was found by investigating one family
at a time over days. vocabulary.sc asks it once: census the calls, subtract everything any fact table
names, rank the remainder by how often it is called and whether it sits in a method that reads
untrusted input. It needs no dataflow, because the structural pass already walks every call.
The harness reads its token set from the SPECS rather than from the TOML, so it measures what the
analyser actually consults: a fact the loader drops is not part of the ontology however neatly it is
written. Ontology sizes today are 152 tokens for JavaScript, 167 for Python, 84 for PHP.
Both carry the constraint the maintainer set: the output is a candidate list, never an ontology. A
name becomes a fact only when someone resolves it to a library API, states the weakness class and
records provenance, because adding a name because a subject called it is how a general-purpose tool
becomes a fixture of its own benchmark.
Also records what the bounded PHP transfer attempt found. At 25 regions the first case now runs to
completion and writes a full artifact in 1,706 s, where at 100 it had exhausted a 2,400 s deadline
with nothing. But 25 of 4,461 regions is 0.6 percent of the population and the declared target was
not among them, so the question was never asked. Task 13.3 therefore depends on 7.3, skipping
arbitration where the evidence says there is nothing to find: raising the budget is already measured
to fail, and making most of the budget unnecessary has not been tried.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The scoping fix looked applied and was not. The same census gap is raised in TWO passes, arbitration
and evidence, and only the first carried the partition it came from, so an unscoped copy kept
demoting every question in the repository. One key in one dict.
What found it was the record, not the code: the census now prints each outcome reason rather than
only its status, and three separate causes had been looking identical as a bare count of unresolved.
Measured on Paid Memberships Pro at 25 regions: questions completed goes from 0 to 120 of 127, with
its 19 findings unchanged. The seven still unresolved are the JavaScript partition own questions,
which is correct, because the four missing files are its files and its answers are the ones that
could be wrong.
Still open, and recorded rather than implied: the push policy _blocking_degradations exempts declared
exclusions but does not yet respect a partition, and a post-hoc diagnostic copy carries no language,
so a JavaScript gap would still block admission of a PHP finding.
1,419 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Membership of GRAPH_INTEGRITY_GAPS should turn on whether a gap can make a reported answer WRONG,
not on whether it cost coverage. An unparsed file may hold the sanitizer that would have cleared a
flow this scan reports, so it belongs there. A flow this tool cannot follow from one language into
another can hide a finding and never invent one, because a PHP sink is sanitised in PHP.
cross_partition_semantics_unresolved therefore moves to DECLARED_EXCLUSION_GAPS. It touches exactly
two places, both of which already honour the distinction: the outcome demotion in scan.py, and
_blocking_degradations in the push policy, whose docstring already says a declared exclusion is not
one of them. The reason stays published in the scope unresolved boundaries either way, so the limit
is still reported; the class decides only whether every answer in the repository is demoted.
This corrects the task note written earlier today, which said not to reclassify on the grounds that
it would claim cross-language flows are out of scope rather than unresolved. That was a misreading
of the class.
PMPro still completes none of its questions, and the measurement now names why: the JavaScript
partition expected 36 files and holds 32, so partition_file_census_incomplete fires and is still an
integrity gap. The four absent files are js/blocks/blocks.build.js, js/jsapi.js, js/select2.min.js
and webpack.config.js -- a build output, a source file, a minified third-party library and a build
config. A bundled select2 costs 4,425 PHP regions their admissibility. Same over-broad blast
radius, third instance of one principle in this thread, and the fix direction is recorded.
The census harness now records each degradation whole rather than only its reason, because a
partition census that reports files missing carries WHICH files and without them the gap is a label
rather than a lead.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
No PHP question is ever marked completed, and PHP was never the variable. partitions.py raises
cross_partition_semantics_unresolved whenever more than one language partition exists, that reason
is a member of GRAPH_INTEGRITY_GAPS, and a graph-integrity gap demotes every completed outcome in
the scan. Admission requires a completed outcome.
The evidence is one column wide. NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is
Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and
completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat own degradation,
vendor_semantics_unresolved, is a DECLARED exclusion rather than an integrity gap, which is exactly
why it does not demote.
This lands on the declared v0.1 target. A WordPress plugin is PHP plus its admin JavaScript, so
every one of them is multi-language, so on the target this tool was built for it can detect defects
and never admit one. Neither subject that currently passes could reveal it, which is why a census
across a third subject was worth running.
Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph
cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task
12.3 already fixed this bug class one level down by making a gap question-owned so it could not
block another question comparison; the same principle applies to a partition boundary. Reclassifying
the reason as a declared exclusion is explicitly the wrong fix, because that would claim
cross-language flows are out of scope when they are unresolved.
Task 13.4 rewritten with the diagnosis, and the milestone amendment now ranks it with M2 rather
than behind it: a narrow JavaScript-only release can proceed without it, the WordPress target
cannot proceed at all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The replay hardcoded 500 regions, which on a 673-file PHP plugin asks 2,526 questions and fits no
ceiling this project sets. The budget is now an argument, defaulting to 500 so every earlier
measurement keeps its shape.
It bounds the RUN, never the repository: the artifact census still reports what was left
unexamined, and a bounded population establishes transfer rather than coverage. Task 13.3.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
PHP is not silent. Paid Memberships Pro establishes findings, including a true positive, and
this project's own instrument is why it looked otherwise.
The census harness ordered its verdict on completion before findings, so a family with eighteen
findings and no completed question reported "asked but no question completed". Findings are now
checked first: a question-outcome count is bookkeeping and cannot hide the thing being censused.
NodeGoat and VAmPI were never affected, because they complete every question and their verdicts
reached the findings check. The scan summary now also carries the total findings and the outcome
status counts, so the two numbers can be seen disagreeing instead of one silently winning.
The second mistake was mine rather than the harness's: only the 500-region runs had been looked
at, which are the ones that time out.
What PHP actually detects on PMPro. At 25 regions, 19 findings in seven minutes with no timeout;
at 100 regions, 43 findings in twelve minutes with no timeout. Among them adminpages/
discountcodes.php:749, where sanitize_text_field($_REQUEST["s"]) reaches
$wpdb->get_results($sqlQuery, OBJECT) through `$sqlQuery .= "WHERE code LIKE '%$s%' "` with no
prepare. That is the class of CVE-2023-23488, which php.toml's own comment names as the reason
those WordPress sanitizers are kept out of the sanitizer list. The prediction and the measurement
agree.
The precision cost beside it is measured too: output encoding reports eighteen findings at 25
regions and the three sampled are all properly escaped, two with esc_attr and one coerced with
intval. Those escapers are absent from a sanitizer list that is flat across families, because
none of them stops SQL injection. Keeping the injection finding costs this family its precision.
The scale limit is roughly one to two seconds per question. 25 and 100 regions complete cleanly;
500 regions asks 2,526 questions, needs more than the 1,800-second ceiling and is killed.
Four things are measured NOT to be the problem: the malformed-CPG bug (fixed in 4.0.625, verified
on the real trigger file), the dataflow overlay (42 s on the full 637-file graph), the engine
version (4.0.625 and 4.0.631 indistinguishable), and the bundled Stripe, Braintree and Twocheckout
SDKs under includes/lib that the vendored list does not cover -- that was my hypothesis for the
cost and the overlay measures 42 s with them and 37 s without, so it is refuted.
What remains is bookkeeping, not detection: questions_completed is zero in every PHP run including
the ones that find things, with cross_partition_semantics_unresolved and
partition_file_census_incomplete present throughout. The admission policy reads those outcomes.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The graph bug is fixed and was never what these runs were hitting. joernio/joern#6281, this
project's own report, is a PHP `global` inside a closure giving a node two AST parents, which made
the dataflow overlay throw and cost the whole graph. Filed against 4.0.623, fixed by #6270 in
v4.0.625, which the image has pinned since before any measurement recorded under task 13.2.
Verified today on the real trigger rather than a transcription of the repro, which does not
reproduce: PMPro's includes/fields.php builds and applies the default overlays cleanly on 4.0.625
and on 4.0.631, files=2 methods=158, no ReachingDefPass assertion.
What remains is runtime. A taint batch of 2,526 questions over the 673-file checkout does not
complete in thirty minutes. Raising the query ceiling six-fold, from 300 to 1,800 seconds, changed
nothing but how long the failure took: query time went from 363 s to 1,874 s, and zero questions
completed in both. The eight-way split retry did not rescue it either, finishing its parts in
about three minutes against a deadline with 3,300 seconds still on it, which is worth its own
look.
The engine pin stays at 4.0.625. An image on 4.0.631, today's release, was built and measured and
is indistinguishable on this subject: 627.7 s against 619.1 s, 363.4 s against 367.3 s of query
time, zero completed in both. There is no demonstrated benefit to weigh against re-verifying every
JavaScript and Python measurement on a new engine. Should that change, the bump is three values
rather than a port: the upstream AstGenRunner.scala is byte-identical in 4.0.631 and its patch
anchor still appears exactly once, so only the jar path, the jar checksum and the source URL tag
move. Both are recorded.
Also closes task 14.5. Neither prototype pollution nor deserialization has a subject that can
answer, each for its own reason. Deserialization's mechanism exists on exactly one available
subject and that subject cannot be queried at all. Prototype pollution has no available subject
containing the mechanism, so its ability to answer is established on a control instead.
1,417 tests pass, eighteen skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat's server.js configures a permissive session at line 78 and disables template
auto-escaping at line 135, both inside the same function. The arbiter returned one verdict per
question, so the escaping defect that task 14.3 added was detected on every run and reported on
none: which of two real defects a contributor saw depended on where in the file it sat.
Configuration now returns one verdict per setting whose value the model could read. The pipeline
already emitted one finding per entailed answer, so this arbiter was the only place still folding
them to one.
Three boundaries are drawn rather than assumed.
- A setting whose value is COMPUTED stays one verdict per region. That band is a question for the
judge rather than a finding, and returning every undecided setting would multiply model calls
for claims the graph did not establish.
- A call carrying two permissive literals is one finding. cors({origin: true, credentials: true})
is one misconfiguration, not two.
- Config verdicts now carry a path:line:function location. Without one they shared the region's
identity and the deduplicator kept a single finding, so the plural answer collapsed straight
back to one. That is the same gap dominance carried until its findings stopped arriving with no
line at all.
Dominance is left answering once per region, and that is settled rather than pending: its claim is
that a region contradicts itself, so the region is the unit and a second verdict would restate the
same inconsistency. The pipeline now says so where it used to call this a known gap.
Measured: NodeGoat's config_secrets goes from one finding to two, server.js:78 for the session and
server.js:135 for the disabled escaping, with the applied marked.setOptions fix between them
staying silent. VAmPI is unchanged and no other family moved.
1,417 tests pass, seventeen skipped; thirteen Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two of NodeGoat's three corroborated claims were correct code. `const { userId } = req.session`
lowers to a temporary and then a field read, so nothing in the graph spells session.userId by the
time the value reaches dao.getByUserId(userId, ...), and the constraint the handler really applies
was invisible.
The arbiter now walks back three hops over the method's own assignments from the names the
operation is GIVEN, and req.session joins the identity sources. The token is only safe because of
that requirement: what discharges an object-level obligation is that the value passed to the
operation comes from the authenticated context, not that a session is mentioned nearby. A handler
that reads the session for logging and keys its query on a path parameter is the bug this family
exists for, and it stays unguarded.
Two exclusions the measurement forced, each caught by a page whose whole purpose is the leak it
demonstrates.
- A callback is not a constraint. Taking identifiers from the whole argument subtree credited
NodeGoat's memo listing with a discharge, because memosDAO.getAllMemos((err, docs) => ... userId
...) reads every memo in the collection and only RENDERS the session's user. Method references
are excluded from the argument names.
- An assignment is not a dominating guard. The session read dominates the query, so that clause
credited it instead and the memo listing went quiet again. An assignment is a read of the
authenticated context, not a check applied to it, so operator calls are excluded there -- the
same exclusion the sink matcher needed. The leak stays reported.
On NodeGoat displayContributions and displayProfile go silent, the allocations insecure direct
object reference, both benefits pages and the memo listing stay, and the login and signup handlers
become honestly unguarded rather than accidentally cleared by a callback's text. Six corroborated
where there were seven, and a better six. VAmPI is unchanged at two entailed findings.
The sibling scope is decided by measurement and kept at the file. On NodeGoat the region's own
file entails two handlers, both public routes and both false, and no true defect at all. Every
handler in the application entails six: four true, and the same two public routes. Both scopes
carry the same false positives, because neither can tell a public route from a protected one --
a semantic question that is already the corroborated band's residual. With the carried rung capped
the scope changes no verdict, only the witness, so widening it would buy a repository-wide walk
per question and no precision. A public-route discriminator would change that; the file scope is
not what holds this family back.
1,415 tests pass, sixteen skipped; thirteen Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
NodeGoat's documented insecure direct object reference reads with
allocationsCol.find(searchCriteria()), and `find` was absent from an operation table holding
findOne, findById, findAll and findByPk. The obligation was missing at the commonest read there
is, so that defect raised nothing at any point in the pipeline.
It was absent for a good reason, and the reason is the argument, not the name. Measured on
wp-statistics, a WordPress plugin that ships jQuery: its assets hold 88 calls named find, of
which 70 pass a jQuery selector, 13 pass a predicate to Array.prototype.find and 5 pass a
selector built at run time by a template literal or a concatenation. None is a collection read.
So the obligation is claimed only where the first argument has the shape of a query document.
The operation fact states which argument with query_document_arg, and the arbiter that can see
argument shapes decides it: no arguments at all, an object literal, or a builder call are
claimed; a method reference, a literal, an operator call and a bare identifier are not. Each
exclusion is one of the measured classes above rather than a guess.
Zero of those 88 calls became an obligated operation, and the zero is measured rather than
empty: the same run's census reports 60 files and 4,552 methods, and the files holding those
calls are in it.
On NodeGoat all three find calls are collection reads and all three are claimed, which raises
the obligation at four more handlers -- the allocations handler that takes userId from req.params
on an authenticated route, the benefits pages whose isAdmin guard is commented out in the route
table, and the memo listing. Seven of fifteen handlers now reach a verdict, all corroborated,
none reported without a judge, and the census is unchanged on both subjects.
A bare identifier is deliberately not claimed: col.find(query) and arr.find(pred) are the same
shape and neither repository measured holds one, so this is a stated false negative rather than
an unmeasured false-positive risk.
1,415 tests pass, sixteen skipped; twelve Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The census measured this family completing 35 questions on NodeGoat with no obligation raised
anywhere: every modelled data operation lives in app/data/*-dao.js, none in a route handler
file, and operations were scoped to the region's own file. The family had nothing to be right or
wrong about.
The obligation now follows the one named call the handler writes, out of the region's file and
no further. How it is followed is the point. jssrc2cpg does not resolve
allocationsDAO.getByUserIdAndThreshold(...) to the DAO body: the callee is an external stub with
no code. That stub's fullName records where the member was written, so the implementation is
reached by the graph's own provenance rather than by a repository-wide name guess, which is the
false positive the file scope was closed against. A control asserts it: a third module defining
a member of the same name that nothing imports is not attributed.
A row is attributed to the region's DECLARED function, which is what lets a question about
displayAllocations match a graph that holds a <lambda>. A carried row is also checked for a guard
at the CALL, because that is where a handler establishes who is asking. The method holding a
carried operation joins the question's context scope, so the module holding the query cannot
change under a claim that depends on it without the change being attributed. The witness names
both sites and the finding is located at the call, which is the line a contributor changed.
A carried obligation corroborates, it does not entail, and that was measured rather than
assumed. Entailing it reported handleSignup, which looks a user up by name to check the name is
free during registration and is public by design, because two authenticated siblings in the same
file pass req.session.userId into the same module. The entailed rung's justification is that one
FILE contradicts itself, same file and same trust context; carried, the compared operations live
in a data-access module that handlers of different trust contexts share. The corroborated band
already asks the right residual: does this operation require a check, or is the route
legitimately public?
Measured on NodeGoat: 15 handlers asked, obligations raised at zero before and six now, three
reaching a verdict, all corroborated, none reported without a judge. VAmPI unchanged at two
entailed findings, because it keeps handler and query in one file and carries nothing.
What this does not reach is recorded, not implied. NodeGoat's documented insecure direct object
reference still raises nothing, because its module reads with allocationsCol.find(...) and find
is not a modelled operation: Array.prototype.find and jQuery's .find are everywhere, so it needs
an argument-shape discriminator and its own measurement (task 14.8). Two of the three corroborated
claims are correct code whose identity read a destructuring spelled away, and the cross-file
contradiction that would entail the real defect needs a sibling-scope decision (task 14.7).
1,415 tests pass, fifteen skipped; eleven Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The census recorded this family completing 44 questions on a Node application and finding
nothing, which is what a client-only vocabulary must do there: all six declared sinks were DOM
sinks. NodeGoat's cross-site scripting is server-side.
Decision: the family asks about a value reaching a rendering call, and states in its own
declaration that it cannot decide template-mediated escaping. Whether res.render is safe depends
on the engine's autoescape setting and on the template's markup, neither of which is at the
call, so res.render is not a sink and the family's limits field says so. A new CWE-79 sink
covers res.write, res.end, res.send and the response.* spellings; res.json stays out because an
application/json body is not a document.
Template auto-escaping is claimed where its evidence is, as a configuration defect:
swig.setDefaults, nunjucks.configure and marked.setOptions are security settings now.
It establishes a finding: app/routes/research.js:25, model_entailed,
`req.query.url + req.query.symbol -> res.write(body)`. The handler fetches a URL the caller
chooses and writes the response body into a text/html response, so the document served from
this origin is the attacker's. The record states how coarse the step is: the path from url to
body runs through the client's callback parameter, which the graph treats as reachable from the
call's arguments, so the conclusion rests on the call's semantics rather than on what the graph
established. Five of seven families now establish findings.
Four defects were found on the way, three of them false positives this change would otherwise
have shipped.
- The permissive-value set was shared across every setting, while a boolean's polarity belongs
to the setting: true leaves the door open in cors({origin: true}) and closes it in
swig.setDefaults({autoescape: true}). Adding the template engines would have reported the fix
as the defect on the line that offers it. ConfigSpec now carries per-setting values, and
Jinja2's Environment moved out of Python's shared list for the same latent reason.
- config.sc matched a dotted setting on its trailing segment alone, so marked.setOptions would
have claimed an editor's setOptions. dominance.sc already required the qualifier.
- taint.sc asked its flow query of every argument including the receiver at index 0. On a
response sink the receiver is the response object, so res.end() -- which takes no data -- was
reported exactly like the res.write(body) two lines above it. Flows now use the real
arguments, as arity already did.
- The arbiter sorted a file's settings by line number as TEXT, so line 126 preceded line 78 and
which setting a file reported depended on the spelling of its line numbers.
Template escaping disabled at server.js:135 is detected, measured model_entailed against the
real query and the real arbiter, and still not reported: the arbiter returns one verdict per
question and that file has an earlier permissive setting. Task 14.6 carries it rather than
hiding it.
1,415 tests pass, thirteen skipped; nine Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The family census recorded this family completing 44 questions on NodeGoat, finding the open
redirect, and having no way to ask about the repository's documented SSRF: needle.get on a
query-string URL at app/routes/research.js:16. A thin vocabulary is a silent false negative.
One data change, no query change. The declared set now covers axios, got, needle, request,
superagent, unirest, undici and fetch as bare module tokens, plus http.get, http.request,
https.get and https.request dotted. Bare is the documented call form of each module and covers
its verb methods too, since a token matches word-bounded against the callee and axios is
bounded inside axios.get; axios.get and axios.post are dropped as subsumed rather than
unwanted. http and https stay dotted because their namespace also holds createServer, and a
bare token would have claimed every server a repository starts.
The request is now reported at app/routes/research.js:16, model_entailed, witness
`req.query.url + req.query.symbol -> needle.get(url, ...)`. This family goes from one finding
to two on NodeGoat, every other family is unchanged, and 299 of 299 questions still complete.
Measured on that graph, needle matches exactly one call and the eleven other clients match
none in first-party or vendored code, so NodeGoat shows the gap closing and cannot show those
clients' noise.
A control asserts both halves. A handler calling axios.get with a literal and needle.get with
req.query.url reports the tainted client alone, and it reads the shipped vocabulary rather than
a copy, so removing the client fails it. The CWE-601 redirect entry is untouched: a
Location-header redirect is value-shaped and stays outside what this family asks.
1,411 tests pass, eleven skipped; seven Joern-gated query tests pass in the image.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Seven vulnerability families are declared and two had been proven. The census asks each of
them, on whole checkouts of NodeGoat and VAmPI, whether it establishes anything at all, and
reports a family that answered and found nothing separately from one that was never asked.
Four establish findings: injection, access_control, config_secrets and untrusted_destination.
Output encoding, path, prototype and deserialization establish nothing on either subject. All
299 NodeGoat and 35 VAmPI questions completed, so the silences are answers.
Three of those silences are diagnosed gaps, not clean code. NodeGoat's documented SSRF calls
needle.get, which the sink vocabulary does not hold. Its XSS is server-rendered, and output
encoding declares only DOM sinks, so no question it can ask could have found it. Its two
access-control defects raise no obligation at all: measured with the real dominance query,
every modelled data operation lives in app/data/*-dao.js and none in a route handler file,
while the family scopes operations to the region's own file. Tasks 14.2 to 14.5 carry each
gap; none is fixed here.
The census also found a false positive and it is fixed. NodeGoat's only prototype report was
the word `set` inside `// set these up in case we have an error case`: the matched node is the
<operator>.assignment holding the entire 1,000-character signup handler, and sinkMatches
searched a node's whole text. It now confines the text clause to the callee and excludes
operator nodes. Fourth instance of this bug class after fgets/gets, resolve/resolveUrl and
user/users. Measured on NodeGoat's non-vendor calls, injection sinks fall 28 to 3, prototype
13 to 2 and untrusted destination 26 to 7, with no true positive lost; req.query had been an
injection sink because `query` is word-bounded inside it. Witnesses now name the sink call
rather than the assignment that contained it.
1,411 tests pass, ten skipped; the six Joern-gated query tests pass in the image, including a
new one that fails on the shape this fixed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Paused at the maintainer's direction to return to vulnerability work, with the
state recorded rather than left implied.
Measured and eliminated: batch size, since 17,700 requests answer in 44 s with a
2.6 MB response; dataflow cost, 50 execution requests in 24 s; call depth 3 with
parameter sources, 24.7 s for 20 requests; argument length, a 16 KB hook table
against a 128 KB single-argument cap; hook callback resolution, one second of
difference with 233 hooks; and build or sharding, 79 s with no shards. Preparation
measures 273.5 s, which is real but a small share of a 3,600-second case.
Every reproduction of the failing 22,125-request batch outside the push path answers
in under a minute. The difference from the production invocation is not identified,
and the exception itself has still never been read: the backend logged a stack
trace's tail, which is frames, and the corrected extractor reports its head, which
on this engine is JVM startup noise.
Four instrument defects were found and fixed along the way, all of the same shape:
a failure detail that discarded the cause, stages that reported no cost, logging
that was never configured so the new stage lines could not emit, and a harness that
buffers stderr until a case ends.
Node and Python transfer stand. PHP stays unestablished.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Retried the PMPro transfer with batch splitting, a 4500-second deadline and a
900-second query ceiling. Stopped after 26 minutes with no case completed and no
verdict, so PHP transfer remains unestablished.
A probe narrowed the cause and corrected my earlier reading of it. The evidence
pass answers all 4,425 PMPro PHP regions in ONE batch in 29.7 seconds with a 681 KB
response, exit 0, and its cost is fixed rather than per request: one request took
40.2 s and 4,425 took 29.7 s. Neither request count nor response size is anywhere
near a limit, so batch size and serialization are not the cause as I first supposed
from a single extrapolated data point.
What fails is the execution pass, which runs reachableByFlows per candidate.
The probe's own scope is recorded with it: it sent evidenceOnly=true and therefore
measured the structural pass, not the failing one. Those numbers would otherwise
read as evidence that PMPro's taint work is cheap, which they are not.
Splitting stays, because it is correct on its own terms and recovers a batch that
fails whole. It is worth noting it also makes an unrecoverable batch slower to
abandon, since each of the eight parts burns its own share of the query ceiling.
M3 remains two of three: Node established in M1b, Python established, PHP blocked
on dataflow cost for a 1,274-file repository.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
All three PMPro comparisons exhausted a 3000-second deadline with a 1200-second
query ceiling and produced no artifact, so there is no target evidence at all. The
engine reports `cpg batch taint failed` twice per case on the 1,274-file tree:
first an exception whose trace runs through ujson array transformation, then a
timeout on the retry.
Budgets tried: 900 s, 1200 s and 3000 s deadlines against 300 s and 1200 s query
ceilings. The pins, source receipts and declared target are all correct, so the
question was asked properly and the engine could not answer it. This is a runtime
blocker on a large repository, not evidence about whether the design transfers to
PHP.
Two harness defects were found on the way, and both had made an earlier run look
like a transfer failure rather than an instrument problem. The manifest labels a
case by what it is, vulnerable or fixed or benign, while the id says what the
comparison does, introduce or repair or benign; filtering on one while keying
expectations by the other ran a single case and reported an all-false verdict. And
the per-query ceiling is taken as the minimum with the remaining transaction time,
so raising only the deadline could never have helped, and twice did not.
M3 is not met. Its exit asks for Node, PHP and Python together: Python is
established, Node was established in M1b, PHP is blocked.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Generalized the replay instrument so the case family is an argument rather than a
hard-coded target, since a harness that only knows one language cannot say whether
the design transfers. Targets, pins and expectations come from the committed inputs,
so Python is asked exactly the way Node was, through the same core, ranker,
comparison and admission.
VAmPI-SQLI models/user_model.py:73:get_user detected, answered
VAmPI-BOLA api_views/users.py:187:update_password detected, answered
VAmPI-BOLA-BOOKS api_views/books.py:51:get_by_title detected, answered
35 of 35 questions complete on every case, nothing reported new, zero admitted
defects.
Two results stand out. The SQL injection is the evidence the milestone spec names
as previously unresolved. And two of the three targets are access control, the
family that could not complete a question on any repository before group 12, so
this is the first time broken object-level authorization has been asked on Python
at all, and it answered on both endpoints.
VAmPI has the same revision on both sides, so the regression mode establishes
detection and completion, never novelty. It asserts that directly: with nothing
changed, a target reported new would be a comparison defect, and none was. That
tests the novelty logic from the opposite side to the NodeGoat case.
Identical revisions cannot show novelty, and VAmPI is inspected teaching data that
cannot qualify production Python support. No capability enabled, no latency claim.
1,407 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured after scoping a registration-only region to its file.
M1a, before group 12 0 of 436 head questions completed
after per-family context 298 of 436
after region attribution 299 of 299, and 299 of 299 on the base
The count falls from 436 to 299 because the mis-attributed function-scoped regions
in a route-registration module are gone, replaced by file-scoped questions the graph
can answer. Nothing is unresolved on either revision.
Detections are unchanged: the vulnerable revision is still new with
operation_absent_from_comparable_base, the fixed twin analyzed and quiet, the
unchanged revision not a regression.
What remains is recorded limits rather than failures: 210 bounded-reachability
boundaries, each owned by the question that reported it, and one vendor exclusion.
Neither blocks a comparison.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Final verification of the three security capability fixes on the pinned revisions.
head questions completed 205 -> 298 of 436
base questions completed 436 of 436
access control completed 0 -> 52
context_projection_unavailable 116 -> 0
context_scope_empty 138 -> 0
Every remaining unresolved question now reports context_function_unresolved and
names the function its file does not define, so nothing is unresolved for a reason
the tool could not explain. That is a region-attribution question for the ranker,
not a limit of what the engine can establish.
Detections are unchanged: the vulnerable revision is still new with
operation_absent_from_comparable_base and every verdict entry holds.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the pinned revisions with each family producing its own change context.
head questions completed 205 -> 298 of 436
context_projection_unavailable 116 -> 0
access_control completed 0 -> 52
Access control is the headline: the family could not complete a question on any
repository before, and now completes every question it is asked. That is a
vulnerability class the tool could not reason about at all.
Detections are unchanged. The vulnerable revision is still new with
operation_absent_from_comparable_base, the fixed twin is analyzed and quiet, the
unchanged revision is not a regression.
The remaining 138 unresolved questions are all context_scope_empty and sit in three
files, none of them access control. They are function-scoped questions whose named
function the query cannot locate in the graph, so the scope it describes is honestly
empty. That is the query-to-operation identity problem M1a recorded at the witness
end, showing up at the scope end, and it is task 12.3 rather than something to patch
by widening a question to its whole file.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Access control and configuration could not complete their change context on any
repository, which left 231 of 436 questions unresolved on the measured Node case
and made two of the three shipped families unable to answer.
The gap was larger than a missing query block. The evidence pass dispatches taint
only, because only taint has an evidence vector to rank by, so these families were
never asked for context at all. A new context pass asks each family with the flag
set.
Each query describes its own scope, which the owner research required and which
matters: copying taint's projection would describe a reachability walk these
queries never perform. Dominance exports the file's methods labelled operation,
guard or sibling, because its claim is that a file contradicts itself. Configuration
exports the file's methods labelled setting or sibling, because it asks what a file
declares. Both report an empty scope explicitly rather than implying one.
A family whose context query answers nothing keeps no context and stays unresolved,
so none can complete on another family's context. A control asserts that directly.
Confirmed on a real NodeGoat graph through the production backend: both families
produce a context summary and seven scoped methods where they previously produced
nothing. All seven are siblings for that file, which is the honest answer since it
declares no access-control operation and no matched setting.
1,405 tests passed, nine skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reverts f04a839 on its own measurement. The seven-class profile rerun recorded
joern_parse_census_incomplete on 12 transactions: the faster build was rejected by
its own census gate and the frontend plus overlay pass ran as well, so those
transactions paid two engine starts instead of one. Every workload kept the same
p50 and the verdict stayed NO-GO.
The cause is an asymmetric gate. The frontend path is not census-gated at build
time, so whatever it produces is accepted, while the joern-parse path was gated and
the same shortfall rejected it. Holding the new path to a standard the old one
never has to meet guarantees the fallback fires. The probe tree and the
materialized push snapshot also differ, so equivalence on the checkout did not
transfer to the real path.
What still stands is the underlying timing fact, measured under the production
retention environment on identical graphs, same 44 files, 579 methods, file names,
overlays, Gruntfile and eighteen test files: 11.0 s against 21.0 s. JavaScript does
pay two engine starts where every other language pays one. A future attempt needs a
gate both paths are held to, or a direct comparison against what the frontend would
produce, and must be measured in the push path rather than on a checkout.
The profile also names the remaining cost plainly: a changed-code transaction
spends its budget before any query runs, head build 25.2 s of a 30 s deadline with
query time 0.0 s. Building one new graph per push is the cost, not querying it.
1,401 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
JavaScript was the only language paying two engine starts: the declared frontend
and then a separate overlay pass. joern-parse applies the overlays itself. On the
pinned Node tree that is 11.0 s against 21.0 s.
The declared frontend stays as the fallback, because it is what the versioned
retention adaptation and the explicit exclusion policy run through, and the census
decides per build: a graph that does not hold every readable declared file is
rebuilt through the frontend rather than trusted. Exclusions keep the frontend path
outright.
Measured under the production retention environment, both paths produce the same 44
files, the same 579 methods, the same file names, the same five overlays, the
Gruntfile and all eighteen first-party test files, with no file present in one
graph and missing from the other.
My first measurement did not establish that. It ran without the retention
environment, both paths dropped to 25 files, and the apparent equivalence was two
runs discarding the same things. The existing control named for that decision
failed against the change built on it, which is exactly what it is for; the
evidence was wrong, not the control. That control now asserts the intent directly:
the faster path is tried first, and a short census rebuilds through the frontend.
1,401 tests passed, nine skipped. A build saving is not the latency gate; the
profile rerun decides that. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The unchanged seven-class profile, three samples each, with the engine session
enabled. Every changed-code workload still times out at the 30-second deadline, and
identical-tip moved the wrong way: p50 rose from 18.5 s to 24.2 s.
The cause is in the artifacts rather than inferred. Thirteen transactions report
session_unavailable:session_startup_timeout, so under the real deadline the session
usually never starts and everything falls back to the disposable path. Startup is 9
to 13 seconds, about a third of the whole budget, and it is spent before any
analysis happens. Where a session did start, on cached-graph transactions whose
remaining work is small, it cost more than it saved.
That is the honest shape of the result: a transaction-owned session amortizes
engine startup across the invocations inside one transaction, and a warm
changed-code push does not have enough of them. It helps a long development-budget
transaction, 326 s to 243 s measured earlier, and cannot help a 30-second one.
Correctness is unchanged throughout: the same three pinned revisions through the
production rules with the session enabled reproduce the M1 verdict exactly.
Group 11 as designed cannot reach the gate, so the dominant remaining cost is named
and the decision is raised rather than worked around. Task 11.6 records the
candidates, since two of them change the product's shape: a session shared across
transactions is a resident local service, which the accepted milestone spec does
not imply, and analysing less is a scope change rather than a speed change. The
30-second and 2-second gates stay unchanged.
Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The leak was the whole cause. With the session closed by the transaction that owns
it, a session-enabled replay of the same pinned revisions under the same budget
keeps its session for the whole run: no session_query_failed appears on either
side, where the previous run reported one per side and fell back to the disposable
path for everything.
Measured on the same pins, instrument and budget, changing only the transport:
vulnerable revision 326.4 s -> 243.3 s
head query time 63.6 s -> 32.0 s
base query time 85.1 s -> 51.2 s
dominance and config ~14 s each -> ~4 s each
Detections are unchanged: witness present, fixed twin analyzed and quiet, unchanged
revision not a regression, kill criterion not triggered. Speed bought by losing a
detection would not be speed.
This is a cold transaction on one teaching repository under a 900-second
development budget. It is not the hook latency gate, which is warm changed-code p95
against the frozen seven-class profile, and that is task 11.5. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
