A taint answer reports traced flows, so an absent row means either that the
operation is not present or that no traced path reached it. Change comparison
needs that distinction to say a change introduced an operation, and reachability
is deliberately bounded, so the two can never be separated from flow rows alone.
That is why the veto guarding the confusion is permanently true on any real
Express application, and why it could be neither discharged nor safely removed.
The taint query now exports one row per operation it enumerated in scope, plus a
completion marker. The marker is what makes an empty inventory meaningful:
without it a scope that was never enumerated and a scope holding no such
operation are indistinguishable.
Comparison separates three outcomes on a changed line: the operation is present
in the base without a traced flow, which is a reachability statement and not
novelty; the operation is absent from an enumerated base, which establishes that
the change introduced it; or the base scope was never enumerated, which stays
explicitly unresolved. A malformed inventory row is not an empty inventory.
The arbiter now skips any row carrying a kind, so an inventory row cannot be
normalized into a flow with an empty source that names no attacker input.
Verified against the real engine on an authored two-revision JavaScript control:
the operation is enumerated on both sides, the base carries it with zero traced
flows, the zero-operation marker appears for an empty scope, and answers and
findings are unchanged. No new sink inference, alias resolver or second source
IR. 1,354 tests passed, nine skipped. Rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Rerunning the three NodeGoat revisions after tasks 10.1-10.3 shows the recorded
view now classifies the case: the vulnerable revision reaches new with
operation_absent_from_comparable_base, and the unchanged revision reaches
unchanged with its base location resolved to the same lines. Both previously
stopped at operation_correspondence_unresolved and witness_identity_unresolved.
The rerun also caught a defect in the repair itself: the new novelty reason was
missing from the admission allowlist, so admission rejected the candidate as
change_unsupported. Fixed with a control.
The production path is still blocked by the last two vetoes, and the measurement
shows neither can be scoped or relaxed as the plan described. A taint answer
reports traced flows, so an absent row means either that the operation does not
exist in the base or that no traced flow reached it.
dynamic_external_or_depth_context_unresolved exists to stop the second being read
as the first, and removing it would turn a wrong absence into a wrong new
finding. It is also computed over every in-scope method, so on any real Express
application it is always true and can never be discharged, which is why the tool
reports nothing on real projects.
The resolution is new evidence rather than a relaxed rule: export the operations
the query actually enumerated, separately from the flows it traced, so absence is
established from the inventory while a traced-flow claim still requires complete
reachability. That is now task 10.4, ahead of the two vetoes it unblocks, and it
adds no detector, alias resolver or second source IR.
1,348 tests passed, nine skipped. No capability enabled; rollout stays NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add group 9 to the approved task plan as the bounded owner amendment release
milestone M1a requires, implement the task-local recorded-veto evaluation flag,
and run the three pinned NodeGoat revisions under it.
The flag re-runs the production comparison and admission over a lifted copy of
the same evidence and records each veto instead of applying it. Recognized
completeness vetoes are lifted and logged; failures never are, because silence
from a failure is not a quiet fixed twin. It is off by default, accepted only on
explicit base/head replay, refused in Git stdin mode, labels every payload it
produces, and admits no defect even when a lifted candidate otherwise would.
M1a outcome, inside its one-week box and with expectations frozen on disk first:
the three entailed <lambda>2 witnesses are present at contributions.js 32-34; the
fixed twin was genuinely analyzed and quiet, answering its target question with
zero rows and no deadline reason while the same findings appear as base-only; the
unchanged revision was analyzed and recorded unchanged. The kill criterion did
not trigger.
Five vetoes fired and were recorded. M1a also corrected the diagnosis: with all
five lifted the detection still stops at operation_correspondence_unresolved,
because a newly introduced operation sits on a changed line and has no base
anchor. The unchanged revision resolves its base location on the same lines and
classifies as unchanged, which isolates correspondence as the mechanism. That
fifth veto was not in the earlier four-veto diagnosis and is now M1b's central
repair.
Diagnosis only. No defect admitted, no capability enabled, no gate changed, no
latency claim; the 900-second development budget cannot measure the hook gate.
Rollout and task 8.3 remain NO-GO.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Two decisions the maintainer made, as rows rather than constants. A vendored tree (vendor/, node_modules/,
third_party/) is somebody else's unit: out of the targets in preprocess and out of the graph through the
frontend's --exclude, which the scan hands to the backend because the CPG seam imports only the standard
library. A test path (tests/, test-*.php, *Test.php) stays in scope -- a test that reaches a sink still
names the sink -- but its regions are not shipped and order after the product's. Measured before the row:
207 of WP Statistics's 357 PHP files were vendored, and 14 of the 67 regions ahead of MW WP Form's CVE
were test files. The position benchmark applies both, so it measures the order the scan spends.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
CVE-2023-23488 is detected at
classes/class.memberorder.php:936:getMemberOrderByCode -- the exact line
verified in the source, in unmodified Paid Memberships Pro 2.9.7.
A two-file slice carrying both ends of the CVE, 2885 lines, loads and
queries in 122 seconds and reports 16 entailed injection findings.
**The other fifteen are not noise.** The file holds 26 `$wpdb` query calls
and **zero** of them use `->prepare`; all 26 interpolate or concatenate a
variable. The engine reports 16 of 26, so it under-reports rather than
over-reports, and every sampled finding is a genuine unprepared query --
`get_row("... WHERE id = '$id'")`, `get_var("... WHERE ID = '" .
$this->user_id . "'")`. CVE-2023-23488 is simply the instance that got an
identifier, because it is reachable unauthenticated; the rest are the same
defect with different reachability.
That answers the question that prompted this: whether the system can find
one real CVE. It can, in production code, at the exact line, in two
minutes, for a cent. What it could not do was survive its own plumbing at
637 files -- four separate quiet failures on this one checkout, all between
the scan and the arbiter, none of them in the analysis.
What this does NOT settle is recorded with it: the full checkout still does
not load and that boundary is unmeasured; whether php2cpg resolves
`new X()` and `$this->method()` is still unknown, since this slice reached
the sink through a same-function flow rather than across the object; and
one CVE on one plugin is not the class-by-class table over several plugins
that 5.8 asks for.
On cutting the input with a model, proposed when size looked like the
blocker: the evidence argues for scoping rather than triage. Two of 637
files found the CVE, task 2.10 already scopes by what a project declares it
ships, and diff-scoping is this spec's thesis. A model choosing what to cut
adds a fifth quiet-failure surface -- drop the file holding the bug and the
scan reports clean with nothing recording why, which is the shape of all
four failures found today. The reverse direction is where it fits: ranking
which regions get the budget, not choosing what to remove.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.7, and it was the wrong question.
**The 117k-line pmpro CPG does not load.** Importing it throws in a
ForkJoinParallelCpgPassWithAccumulator during overlay application, so every
query against it returns nothing. It is not a call-graph question at all.
Isolated by running the ARBITER against two prebuilt CPGs rather than
writing a sixth ad-hoc Scala probe, four of which had already failed on my
own mistakes:
small wpdb CPG get_order_by_code -> 1 model_entailed
pmpro CPG getMemberOrderByCode -> 0
Both have their source and sink in ONE function, both were asked with
parameter sources and depth 3, same arbiter and same spec. The CPG is the
only difference.
That clears the facts, which is worth stating: `$wpdb` sinks work, the
parameter-source rule works, and a PHP parameter reaching `$wpdb->get_var`
through a concat is entailed. Whether php2cpg resolves `new X()` and
`$this->method()` is still unmeasured -- it cannot be measured on a CPG
that will not load -- and is recorded as 5.10 along with finding the size
boundary.
**And the defect underneath it, which is the fourth quiet failure found on
this one checkout.** `query_batch` returned `{}` for a failure and `{}` for
a genuine empty answer, so a caller could not tell "could not ask" from
"nothing found". Every query on this scan failed at load and it reported
500 regions examined, no findings, and no degradation: a vulnerable plugin,
clean.
A failed batch now returns None, unparseable output counts as a failure
rather than an empty answer, and the driver records a `query_failed`
degradation naming the kind and the request count.
This is the failure mode the rung ladder structurally cannot catch. Every
other honesty mechanism here labels a VERDICT -- suspicion,
regions_unjudged, regions_unasked, stated family limits -- and when no
verdict is produced there is nothing to label.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Reporting the result rather than the intention.
Paid Memberships Pro 2.9.7 with CVE-2023-23488 is pinned, verified in the
code, and scans cleanly: 637 files, 117k lines, 1577 regions, build 47s,
query 74s, 140s total, no degradations. **The CVE is not detected.**
Both ends are inside the budget -- the REST handler at position 290 as a
recognised entry point, the `$wpdb->get_var` sink at 472 -- and the taint
query runs for 23 seconds. So the gap is the middle: the flow crosses
`new MemberOrder( $code )` into a constructor and then
`$this->getMemberOrderByCode( $code )`, both dynamic in PHP. Whether
php2cpg resolves either into a callee edge is UNVERIFIED; two attempts to
query the call graph failed on script errors and were abandoned rather than
guessed at. Recorded as 5.7, with the answer required before anything is
concluded about the fact tables.
Three engine defects had to be fixed before the scan could produce even
this result, and each had previously reported the plugin clean for a
different reason:
1577 regions, 0 scanned joern-parse failed where php2cpg succeeded
500 scanned, taint 0.05s a 1.49MB batch over Linux's 128KB argv cap
500 scanned, taint 21.7s the entry point was not recognised as one
The middle one is the one to remember. Every honesty mechanism in this
codebase labels a verdict -- suspicion, regions_unjudged, regions_unasked,
stated family limits -- and there no verdict was produced to label, so a
vulnerable plugin was reported clean with every counter agreeing.
What is established: the WordPress engine path works end to end. What is
not: detection on a real plugin. One checkout that does not find its CVE is
not yet evidence either way about the facts, and 5.7 has to answer whether
the call graph can carry the flow at all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.8. VAmPI's last two false positives were register_user
and login_user, both declared open by its OpenAPI spec: the arbiter was
reporting a missing guard on endpoints whose contract says there is
nothing to miss.
**The fix keys on provenance, not on the value, and that distinction is
the whole task.** The mapper says "public" in two very different
situations. An OpenAPI operation with no `security` block is DECLARED
open, because the spec is a complete contract. A Flask handler with no
`@login_required` is "public" only because nothing was found -- which is
precisely the bug this family exists to report. Suppressing on the value
alone would have silenced the detector on its primary case, so
EntryPointRecord now carries `access_declared` and only a declaration
removes the obligation.
WordPress supplied the second source of declared access on the same day --
wp_ajax_nopriv_ for a logged-out handler, wp_ajax_ for a logged-in one,
permission_callback => __return_true for an open REST route. A rule resting
on one framework's convention would be a special case; two make it a shape.
vampi entailed 6 -> 4
access-control precision 2 of 4 -> 2 of 2
known vulnerabilities 3 of 3 still entailed
access_control offered to 6 of 14 handlers, exactly the six the spec
declares authenticated
**And the guard that decides whether it is a real improvement:** the
access-control pair corpus, which nobody tuned this against, is unmoved --
pair_correct 3 and leak 1 across all nine, pair_correct 2 and leak 0.0%
across the seven from projects unrelated to VAmPI, identical to task 2.6.
The precision was not bought by silencing the family. Pair excerpts carry
no route manifest and no hook, so nothing in them is declared open and the
rule correctly does not fire.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.17, completing Req 5.5.
Task 2.11 made static pattern matches proportionate and left the model
layer's own suspicions alone, reasoning that a proposer had reasoned about
them. That does not survive inspection. A model suspicion is what the
enumerator proposed at a site the graph could NOT decide, affirmed by a
judge with no coverage there -- and its own rationale says exactly that:
"the model resolved no flow here; its sink table may not cover this API".
It is a pattern match with an opinion attached, and it reads like a
reasoned claim only because a model produced it.
vampi 60 findings, 61 sections -> 26, report 1085 lines -> 446
libpng 198 findings, 39 sections -> 30
**What it costs, which is the number that decides whether the rule is
right.** For each known vulnerability the recipe declares, take every
finding row at its location and ask whether any survives the rule. All
three of vampi's -- the SQL injection and both documented BOLAs -- are
still found at model_entailed and all three survive. Nothing known is
hidden. Three vulnerabilities in one repository is also a statement about
how small the evidence base is, and the artifact says so.
The obligations checker sits at `suspicion` too and is NOT grouped: its
output comes from a structural analysis, an obligated operation with no
discharging guard, rather than from a guess. An earlier attempt at this
rule filtered on rung alone and would have collapsed it; six tests failed
and they were right to.
"Pattern matches" is now "Proposed, not corroborated", since it holds two
shapes. Model rows group by FAMILY rather than by rule, because their rule
is always `model`.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.5 recorded that php2cpg emits a CPG containing only one PHP file --
the last alphabetically -- and called it a blocker for the WordPress
target. Rerun with a native php-cli, as asked, and it is not true.
php 8.3.6, no shim, same joern 4.0.623, same two files:
files: plugin.php, vuln.php
methods: mp_fetch_report, mp_save_report, mp_render_box, mp_helper,
get_user, safe_user, show, run
Both files, all eight methods, and no PHP-Parser banner warning at all --
the banner that looked like the thread to pull was a symptom of the same
thing and does not occur natively.
The cause was the workaround. This machine has no PHP interpreter, so the
shim ran each `php` invocation in its own container; php2cpg drives
PHP-Parser once per file, and that per-invocation isolation kept only one
file's JSON.
**The lesson is worth more than the fix.** Two independent reproductions
agreed, and both were measured through the same broken instrument. A
consistent result is not a correct one when every measurement shares an
apparatus -- the same trap as a corpus that cannot see the defect it is
scoring, which is this session's other recurring finding.
And with the instrument fixed, 5.4 is demonstrably done. A two-file PHP
tree yields 10 function-scoped regions and 7 entailed findings in 15
seconds: SQL injection in both WordPress handlers, XSS in the shortcode
renderer, and all three of the original probe's. safe_user is correctly
absent, its mysqli_real_escape_string dropping it to corroborated. The
wp_ajax_nopriv_ handler ranks 1.0 and the plain helper 0.3.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.4. Both halves, and the second found a frontend blocker.
**One verdict per region was right for a pair and wrong for a file.** A
pair is one labelled function with one bug, so one answer is correct. A
region spanning a PHP file is not: the probe's SQL injection on line 5 and
command injection on line 13 produced ONE injection verdict, `system` won
it on flow length, and the SQL injection went unreported by construction
rather than by any failure of the analysis.
`taint.verdicts()` now returns one verdict per distinct sink site,
strongest first, and its first element is exactly what `verdict()`
returns -- so the pair path and every committed measurement resting on it
are unchanged, and the gates are byte-identical. Both entry points share
`_usable_flows` so neither can drift from the other.
php probe 2 of 3 -> 3 of 3
vampi 6 entailed, unchanged, now with lines on access-control
findings from task 2.7 (users.py:187, books.py:51 -- the two
documented BOLAs at their exact lines)
libpng 1 -> 2 entailed; the second was hidden by the collapse.
Neither is validated and both are in contrib/.
**WordPress registers handlers rather than decorating them, and the hook
name is the declaration.** `wp_ajax_nopriv_*` is WordPress saying
logged-out callers reach this, so it is public; `wp_ajax_*` fires only for
logged-in ones; `permission_callback => __return_true` declares a REST
route open. That is the declared-provenance signal task 2.8 needs,
arriving from a second framework. Plus every top-level PHP function as its
own region, without which a file is one region however many functions it
holds: 1 file-level region became 10 function-scoped ones.
**And a blocker, recorded as 5.5.** php2cpg emits a CPG containing only
ONE PHP file, the last alphabetically. Reproduced twice on independent
samples; both files parse cleanly under `php -l` and under PHP-Parser
directly, and joern-parse reports success either way. php2cpg logs "Found
warning in PHP-Parser JSON output" naming the survivor, and that warning is
PHP-Parser's own `====> File X:` banner appearing in what php2cpg expects
to be pure JSON.
Not isolated yet: this machine has no PHP interpreter, so php2cpg was
driven through a container shim. The banner comes from PHP-Parser rather
than the shim, which makes a shim-only cause unlikely but not excluded, and
rerunning with the native php-cli in the shipped image is the first step.
It blocks WordPress outright -- a plugin is many files, and a scan that
silently covers one of them and reports success is the quiet failure Req
5.6 was approved for.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.1, the first step toward the v0.1 WordPress target.
Written against a PHP CPG rather than against PHP source, because the two
do not look alike. Verified with joern-parse 4.0.623 / php2cpg on a probe:
$_GET["name"] -> call `<operator>.indexAccess`, CODE $_GET["name"]
mysqli_query($conn, $sql) -> call NAMED mysqli_query
echo $_GET["msg"] -> call NAMED echo
So sources match on code and sinks match on name, and every entry the
probe did not exercise is marked unverified in the file -- `include` and
`require` among them, which is PHP's most characteristic path bug and the
first thing to check on a real checkout.
Five families derive (injection, output_encoding, path, deserialization,
untrusted_destination) with no sink that is its own sanitizer -- asserted
by a test, because that bug class has now appeared three times in three
matchers. `php` is added to the region model's languages, without which
the facts would have reached nothing.
The arbiter decides 4 of 4 on the probe: the SQL concat entails, its
`mysqli_real_escape_string` twin drops to corroborated, `echo` XSS entails
and `system` command injection entails.
**A whole scan finds 2 of those 3, and the gap is the finding.** PHP files
yield FILE-LEVEL regions because no PHP entry points are mapped, and the
taint arbiter returns one verdict per region per family -- so a file-level
region reports at most one injection finding however many it holds.
`system` won on flow length and the SQL injection went unreported. That is
the mirror image of task 2.9: that one is many regions collapsing to one
site, this is many sites collapsing to one verdict, and it bites hardest
exactly where PHP needs it not to, since a WordPress file is large and has
no decorator for the mapper to find.
Recorded as task 5.4 with WordPress's registration model (add_action,
add_filter, admin-ajax, REST routes), which the mapper already handles in
principle for other frameworks.
A 13-line probe is not a WordPress result. What is established is that the
seam works end to end for PHP; everything about scale and coverage is
unmeasured, and there are still zero PHP pairs in a corpus of 263 python,
351 c_cpp, 47 javascript and 49 java.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting
that `memcpy` and `strcpy` are PRESENT in a library that uses them
correctly throughout. That is the noise developers have learned to skip in
every other tool, and shipping it costs the findings beside it their
credibility.
**Filtering on rung was the obvious rule and it was wrong.** The
obligations checker emits at `suspicion` too, so hiding that rung would
have hidden an entire detector -- six tests failed and they were right to.
The discriminator is evidence_level: on libpng 163 of 198 rows are
`static_corroboration`, on vampi 3 of 60 are, and the model layer and the
obligations checker carry `suspicion` there instead. That is exactly the
distinction the ladder exists to make -- an undecided CLAIM about a site
is not an observation that an API exists.
So a pattern-only rule is grouped once it has more than three sites, and
keeps its full sections under that. One `python-flask-debug` is a finding a
contributor can act on and must not be summarised away; fifty-four
`c-unsafe-copy` are a property of the codebase.
libpng 198 findings, 198 sections -> 39, 161 grouped into three lines
vampi 60 findings, 61 sections -> 61, nothing grouped
Counted, never dropped: findings.json stays complete because it is the
record a baseline is compared against, and the grouped section says the
model reached none of these and that its silence means no fact table
covered the site rather than that the site is safe.
What it does not fix, recorded rather than glossed: vampi still emits 61
sections for a 520-line application, 51 of them reasoned claims this rule
does not touch. It fixes pattern noise, not volume.
Also records what group 2 established about the three corpus asks, so none
is started on a wrong assumption. Joern ships php2cpg but our fact tables
have zero PHP families, so WordPress would parse and produce nothing --
fact tables first, corpus second. An older vulnerable libpng will not help
for a reason no version can fix. Vibe-code repositories are the best fit
and the cheapest route to a second true-positive checkout, because Python
and JavaScript are the two languages whose fact tables are complete.
And Group 8, for per-repository state a developer owns: a committed
dismissal file the tool reads and never writes, dismissals as a split
evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION --
the judge and residual prompts -- never to the fact tables. That boundary
is the whole design: a prompt is how we ask, a fact table is what we
claim, and the closed-loop leak measured on 2026-09-05 is why the
distinction is not negotiable.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.12. c/injection and c/memory carried zero sanitizers,
so no C flow could ever be discharged: every `argv -> strcpy` entailed
however carefully it was bounded.
Which sinks can be discharged this way is read from the sink fact's own
CWE rather than declared: CWE-121 is a stack buffer overflow, so
gets/sprintf/strcat/strcpy qualify. It is the only overflow CWE in the
fact tables and it is C-only, so no Python or JavaScript sink is touched
and the pair corpus is unaffected -- gates byte-identical.
Not a sanitizer list, because `strlen` appears in the guarded and the
unguarded case alike; the discharge is a guard, not a call.
**The first implementation required the bound to DOMINATE the sink, and it
found nothing.** libpng guards `strcpy(outname+len, ".png")` with
`(len = strlen(inname)) > 250`, but that check sits on an else-if chain
whose sibling branch skips it entirely, and the protection reaches the
sink through an `error` flag: `++error` in the guarded branch, `if
(!error)` around the copy. Joern's dominator set for that sink has 89
calls and correctly excludes the bound; `controlledBy` returns none
either. Verified against the CPG before the rule was changed rather than
after it failed.
So the bound is searched in the method, and the consequence is
corroborated rather than silence. That is the honest rung: a bound on this
variable means the graph has NOT shown the copy is unguarded, and it has
not shown it is guarded either -- off-by-one being how bounds fail. The
judge is asked. Proving the flag-mediated case needs path-sensitive
reasoning this arbiter does not do.
libpng entailed 3 -> 1
The survivor is also a false positive: `strcpy(output + len, ".icc")` in
iccfrompng.c, where `output = malloc(len + 5)` makes it fit exactly. That
is a different discharge shape -- allocated to size, not bounds-checked --
and it is recorded as 2.13 rather than chased. Both shapes were found on
one repository, and extending a rule until that repository is clean is the
overfitting Req 8 exists to prevent; 2.13 requires a second C checkout
with a genuine CWE-121 so the change can be measured for what it silences
as well as what it clears.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.10. Every one of libpng's twenty-five entry-point
regions was a `main()` -- twenty-three under contrib/, plus example.c and
pngtest.c -- and those regions carried the fifty interprocedural taint
requests that dominated a 300s query phase. A library has no entry points,
so entry-point-driven analysis had nothing to anchor to except its own
example programs.
The task said to derive this rather than hardcode `contrib/`, and a project
already declares it. Makefile.am names installed targets in
`lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES`
names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own
word for "not installed" -- exactly the distinction wanted. Taking every
`_SOURCES` would have called libpng's whole test suite shipped code, since
contrib/libtests declares its sources the same way the library does.
interprocedural regions 25 -> 1
taint query 300.1s -> 125.0s
total query 311.9s -> 138.5s
regions 121 -> 121, unchanged by design
The surviving interprocedural region is contrib/tools/pngfix.c:main, which
libpng genuinely installs.
Two rules keep it honest. **Silence is not exclusion**: a project with no
build file this can read gets None back and nothing changes, so vampi's
numbers are untouched. **Not shipped is not unscanned**: those regions are
still arbitrated function-locally, because a contributor editing a test
must still get an answer -- what they lose is the expensive treatment and
first claim on the budget.
Entailed findings went 0 -> 3, and all three are false positives at
contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe:
outname is char[256] and line 325 guards with
`if ((len = strlen(inname)) > 250)`. They were not introduced so much as
uncovered -- a different callDepth meant a different sink set, and the 26
fgets false positives were masking them.
The structural cause is recorded as 2.12: c/injection and c/memory carry
ZERO sanitizers, so no C flow can ever be discharged and every
`argv -> strcpy` entails however carefully it is bounded. The discharge
here is not a call but a guard that dominates the sink, which is precisely
what dominance.sc already does for access control and is wired to that one
family.
Also adds Group 7, five tasks for stopping the same false positive twice.
Two of today's three false-positive classes were one bug in three
matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in
`fgets` -- each fixed locally and each next one surviving because nobody
carried the fix across. A model may group counter-examples by cause and
draft an edit; only a split-respecting measurement may adopt one, because
the predecessor's improve lever admitted shapes taught by the holdout pairs
they then recovered and turned a -0.059 Youden into +0.059.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform.
89,082 lines of C across 93 files, under a 2GB heap cap:
CPG build 16.1s (25s cold)
Joern queries 250.0s
arbitration 122.0s (105 calls, $0.0122)
whole scan 446s
peak RSS 911MB
regions 121, all arbitrated, 32 unasked
CVE-2025-66293 out_of_scope, as the recipe declares
Memory is not the constraint a fleet runner would hit: 911MB leaves room
for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen
while building this was concurrent JVMs on one desktop, not repository
size.
**Server mode: no.** The query phase splits taint 243.3s / config 10.7s
over two invocations, and fixed overhead measured on the same CPG is ~13s
per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL
evaluation: a warm server removes 6% of the scan and none of the part that
costs. The driver now records query seconds per kind, because a single
total cannot tell startup from evaluation and the two answer this
oppositely.
The small repository pointed the other way. vampi's 34.3s over three kinds
is roughly three startups and almost no evaluation, which made a warm JVM
look worth having; the ratio inverts with size. That inversion is why the
task was sequenced after a real repository rather than decided on a
520-line file.
On whether one whole-repository CPG is the only shape for a horizontal
fleet: no, and it is also not where the pressure is. The build
parallelises and costs 16s; the query does not shard and costs 250s, so
splitting a repository across runners divides the cheap phase -- and after
2.4 a shard boundary hides exactly the cross-module flows that matter.
Scoping to a contributor's diff cuts the expensive phase instead. 911MB
for 89k lines extrapolates to roughly 5GB at a million lines, which would
exceed a standard runner, so the ceiling is real and further out than this
checkout can locate.
Two defects recorded rather than fixed, both invisible at vampi scale:
2.9, all 26 entailed findings are the same site reported 26 times, because
regions that share a sink each emit their own; and 2.10, 22 of the 30
highest-ranked regions are libpng's `contrib/` example programs rather than
the library.
Also ignores Joern's `workspace/`, which a maintainer querying a CPG by
hand from the repository root will create.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.6. Access control entailed 7 of 9 VAmPI handlers,
including three that do authorize and three the OpenAPI spec declares
public. Five separate defects, each invisible to a corpus of one-function
excerpts and each found by asking the query what it actually matched.
**A trailing segment is not a match.** `query.get` was matched by its last
segment, so every `.get(` in the repository became a protected database
operation -- `request.headers.get('Authorization')` and
`request_data.get('username')` included. Four of the seven false positives
came from this, and so did the apparent detection of the known BOLA: it was
entailed because fetching the Authorization header had no dominating guard,
not because of the defect. A qualified token now requires its qualifier.
**Configuration is not authorization.** dominance_specs took every
obligation fact. `security_setting` operations are discharged by a VALUE,
which config_specs already claims them for; pooled in here they made
`app.run(...)` an operation requiring an authorization guard and put
thirteen tokens in BOTH lists -- an obligation that is its own discharge,
the same shape as the taint sink that used to sanitize itself.
**Input validation is not authorization.** `jsonschema.validate` was
counted as a guard, so handlers with no authorization check at all read as
guarded. No operation in the table requires validated_input.
**A guard discharges the obligation it is a guard of.** The largest one.
`orm-read` requires an identity_constraint or an ownership_check;
`token_validator` is a path_guard. Flattened into one list, authenticating
the caller discharged an object-level obligation -- a handler that
authenticates and then looks the record up by a PATH parameter read as
fully guarded. The facts state that relation and the spec was discarding
it; it is now carried and the engine asks per operation.
**A word boundary needs a word.** Found while fixing the above. Guard
matching wrapped every token in `(?<![A-Za-z0-9_])...(?![A-Za-z0-9_])`.
The identity token `['sub']` begins with `[`, and the character before it
in `resp['sub']` is `p` -- so the assertion could never hold. Every
identity token starting with punctuation was unmatchable, and the identity
constraint that separates a correct lookup from an IDOR was invisible.
Also: dominance rows were repository-wide, so the model's unguarded
`delete_user` was attributed to the API handler of the same name. Rows are
scoped to the region's file, which is what the rung's own justification
says -- the evidence is a file contradicting itself.
Measured against VAmPI's OpenAPI `security` blocks, which declare per
operation whether auth is required, and agree with the code:
vampi precision 28.6% (2/7) -> 50.0% (2/4)
known found 3 of 3, all model_entailed
And on the labelled access-control pairs, which nobody tuned this against:
all 9 pairs pair_correct 3 -> 3, leak 44.4% -> 11.1%
7 non-VAmPI pair_correct 2 -> 2, leak 42.9% -> 0.0%
Every detection lost was on a pair whose FIXED side was flagged too, which
scores nothing. No correct detection was traded for the precision.
get_by_title was flagged before it was in the recipe: its code carries the
comment "# Broken Object Level Authorization" and the README lists it, so a
real finding was being scored as a false positive. It is now the recipe's
third known vulnerability.
Both remaining false positives are endpoints the spec declares public.
Recorded as 2.8, with the reason the mapper's access_level cannot simply be
trusted: for decorator frameworks "public" means "no decorator found",
which is the bug itself.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.5. VAmPI's `host='0.0.0.0'` is now reported exactly
once, at app.py:17:<module>. Before task 2.2 it was reported eight times
in eight files that do not contain it; after 2.2 it was reported nowhere.
Two causes, one shape: nothing could ask about module-level code.
**The main-guard marker is not a function.** `__main__` is what the mapper
emits for `if __name__ == '__main__':`. Sent to the engine as a function
name it matched nothing, because a CPG calls a file's top-level code
`<module>` -- verified against the VAmPI CPG rather than assumed. The
region model translates the marker, which is the one place that turns what
the mapper meant into what the scanner asks.
**A function-scoped file had no region for its module body.** Settings,
route registration and globals live in no function, and every region's
query filters by function name. A file-level region beside the handlers
would report each handler's findings a second time, which is why the
earlier rule dropped it as "a second full-file pass". A module-scope region
PARTITIONS the file instead of overlapping it. That only became cheap after
batching: before it, a region was a JVM.
Two things this needed to be correct rather than merely present:
- A module method spans its whole file, so the taint query's line-range
containment made `<module>` the parent of every function in it. Under
module scope, membership is direct -- otherwise the module region reports
every handler's sinks again, which is the duplication the old rule was
avoiding.
- A module body is not a handler. The marker-derived region inherited the
handler families and entailed a missing authorization check on the main
guard: a finding with no possible remediation. Caught by the first
repository run that had the region, and now covered by a test.
bind address 8 wrong files -> 0 -> 1, at app.py:17:<module>
regions 22 -> 25
entailed 8 -> 9
query time 37.4s -> 41.3s
peak RSS 827MB -> 823MB
cost $0.0067 -> $0.0068
Three more regions for four seconds and no more model calls: the budget is
a total and was already spent, so regions_unasked rose 17 -> 20 -- those
regions got the free pass and are reported as unasked, not as clean.
`<module>` is Joern's Python name for a module body; JavaScript uses
`:program`, so this is verified for Python only.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.4, closing the no-go task 2.2 recorded.
VAmPI's SQL injection is a request parameter reaching `get_by_username`
in api_views/users.py, passed to `User.get_user` in models/user_model.py,
interpolated into a query string and executed there. Source and sink are
in different functions and different modules. Regions were per-function,
so neither region saw both ends and the engine was silent on the most
ordinary shape a real injection bug has.
`reachableByFlows` was never the limitation -- it follows calls already.
The limitation was the QUESTION. Sinks were filtered to the labeled
function's own body, so a crossing flow had no sink to end at.
Three changes, each with a reason it is sound rather than convenient:
**An entry point is the trust boundary.** parameter_sources was off for
repository scans because a repository's parameters are not all attacker
input. An entry point's are, by definition -- that is what makes it an
entry point. The driver turns it on only for regions the mapper found and
named, and leaves it off for walker fallbacks and for named functions that
are not entry points.
**Sinks may sit below the handler.** taint.sc admits sinks in methods
reachable from the labeled method through the call graph, bounded at
ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is
where these bugs live. Unbounded would make the question about most of the
repository rather than about this entry point, and make it unaffordable.
**The finding names the sink, not the asker.** A crossing flow ends in
another file and another function, so Verdict carries the evidence's own
`path:line:function` and the site uses it. Reporting
`models/user_model.py:73:get_by_username` would name a place that does not
exist: line 73 is inside get_user.
Also: labeled methods are now resolved by name AND file. `get_user` exists
in more than one module of a real repository, which a corpus of one-file
excerpts could never surface.
Measured on the same checkout and budget as 2.2:
VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user
known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now
entailed 7 -> 8
query time 40.9s -> 37.4s
peak RSS 869MB -> 827MB
cost $0.0067, unchanged
Asking the bigger question cost nothing measurable here: file scoping and
call-graph following pull in opposite directions and roughly cancel at this
size. That is one 520-line repository, and VAmPI is now the repository this
change was developed against -- the exact overfitting shape Req 8 exists to
catch. The depth bound and the entry-point rule mean nothing general until
a checkout nobody tuned against says so.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.
The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.
**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.
**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.
**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.
**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.
Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.
Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The pair corpus scores excerpts: a vulnerable function and its fixed twin,
handed to the engine with the answer already narrowed down. Nothing in it
exercises a repository -- region ranking, the budget, one CPG over
thousands of files, or whether a verdict that holds on fifteen lines still
holds when the source, the sink and the guard live in three modules.
benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char
commit SHA, licence, and every known vulnerability with its file, function
and line. `ousast repos` lists them; `--fetch` materialises one; CI does
neither, and conftest clears the switch and redirects the cache so a
developer with warm checkouts cannot read them by accident.
Three rules the loader enforces rather than documents:
- A branch or tag is rejected. Either would make the same recipe mean
different code next month while the baseline it produced kept the old
numbers.
- `in_scope` must be stated per vulnerability, never defaulted. It decides
whether "not found" is a miss or a non-question, which is the most
important thing about the result.
- A recipe that declares `detection` must name something in scope to
detect.
`ousast repos` does not trust a recipe either: against a materialised
checkout it checks HEAD against the pin, that the licence file is there,
and that each declared function appears above its declared line. That
earned its place immediately -- the first libpng recipe named LICENSE.md,
which is the filename at HEAD today and not at the pinned commit, where it
is still LICENSE.
Two checkouts, verified against the real repositories:
- vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are
already in the pair corpus, including the update_password miss diagnosed
earlier, so a whole-repository run says directly whether a verdict
survives the move off the excerpt. Its routes register through an
OpenAPI operationId, so it exercises registration-based entry-point
mapping rather than the decorator path.
- libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at
the parent of the commit that added the missing clamp. Its CVE is marked
out of scope: an out-of-bounds read is not a family this engine
arbitrates, and recording "not found" against it as a miss is the
easiest way to make a measurement lie.
Nothing is vendored; the recipe records the licence so an artifact can say
what it ran against.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
