Two decisions the maintainer made, as rows rather than constants. A vendored tree (vendor/, node_modules/,
third_party/) is somebody else's unit: out of the targets in preprocess and out of the graph through the
frontend's --exclude, which the scan hands to the backend because the CPG seam imports only the standard
library. A test path (tests/, test-*.php, *Test.php) stays in scope -- a test that reaches a sink still
names the sink -- but its regions are not shipped and order after the product's. Measured before the row:
207 of WP Statistics's 357 PHP files were vendored, and 14 of the 67 regions ahead of MW WP Form's CVE
were test files. The position benchmark applies both, so it measures the order the scan spends.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
CVE-2023-23488 is detected at
classes/class.memberorder.php:936:getMemberOrderByCode -- the exact line
verified in the source, in unmodified Paid Memberships Pro 2.9.7.
A two-file slice carrying both ends of the CVE, 2885 lines, loads and
queries in 122 seconds and reports 16 entailed injection findings.
**The other fifteen are not noise.** The file holds 26 `$wpdb` query calls
and **zero** of them use `->prepare`; all 26 interpolate or concatenate a
variable. The engine reports 16 of 26, so it under-reports rather than
over-reports, and every sampled finding is a genuine unprepared query --
`get_row("... WHERE id = '$id'")`, `get_var("... WHERE ID = '" .
$this->user_id . "'")`. CVE-2023-23488 is simply the instance that got an
identifier, because it is reachable unauthenticated; the rest are the same
defect with different reachability.
That answers the question that prompted this: whether the system can find
one real CVE. It can, in production code, at the exact line, in two
minutes, for a cent. What it could not do was survive its own plumbing at
637 files -- four separate quiet failures on this one checkout, all between
the scan and the arbiter, none of them in the analysis.
What this does NOT settle is recorded with it: the full checkout still does
not load and that boundary is unmeasured; whether php2cpg resolves
`new X()` and `$this->method()` is still unknown, since this slice reached
the sink through a same-function flow rather than across the object; and
one CVE on one plugin is not the class-by-class table over several plugins
that 5.8 asks for.
On cutting the input with a model, proposed when size looked like the
blocker: the evidence argues for scoping rather than triage. Two of 637
files found the CVE, task 2.10 already scopes by what a project declares it
ships, and diff-scoping is this spec's thesis. A model choosing what to cut
adds a fifth quiet-failure surface -- drop the file holding the bug and the
scan reports clean with nothing recording why, which is the shape of all
four failures found today. The reverse direction is where it fits: ranking
which regions get the budget, not choosing what to remove.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.7, and it was the wrong question.
**The 117k-line pmpro CPG does not load.** Importing it throws in a
ForkJoinParallelCpgPassWithAccumulator during overlay application, so every
query against it returns nothing. It is not a call-graph question at all.
Isolated by running the ARBITER against two prebuilt CPGs rather than
writing a sixth ad-hoc Scala probe, four of which had already failed on my
own mistakes:
small wpdb CPG get_order_by_code -> 1 model_entailed
pmpro CPG getMemberOrderByCode -> 0
Both have their source and sink in ONE function, both were asked with
parameter sources and depth 3, same arbiter and same spec. The CPG is the
only difference.
That clears the facts, which is worth stating: `$wpdb` sinks work, the
parameter-source rule works, and a PHP parameter reaching `$wpdb->get_var`
through a concat is entailed. Whether php2cpg resolves `new X()` and
`$this->method()` is still unmeasured -- it cannot be measured on a CPG
that will not load -- and is recorded as 5.10 along with finding the size
boundary.
**And the defect underneath it, which is the fourth quiet failure found on
this one checkout.** `query_batch` returned `{}` for a failure and `{}` for
a genuine empty answer, so a caller could not tell "could not ask" from
"nothing found". Every query on this scan failed at load and it reported
500 regions examined, no findings, and no degradation: a vulnerable plugin,
clean.
A failed batch now returns None, unparseable output counts as a failure
rather than an empty answer, and the driver records a `query_failed`
degradation naming the kind and the request count.
This is the failure mode the rung ladder structurally cannot catch. Every
other honesty mechanism here labels a VERDICT -- suspicion,
regions_unjudged, regions_unasked, stated family limits -- and when no
verdict is produced there is nothing to label.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Reporting the result rather than the intention.
Paid Memberships Pro 2.9.7 with CVE-2023-23488 is pinned, verified in the
code, and scans cleanly: 637 files, 117k lines, 1577 regions, build 47s,
query 74s, 140s total, no degradations. **The CVE is not detected.**
Both ends are inside the budget -- the REST handler at position 290 as a
recognised entry point, the `$wpdb->get_var` sink at 472 -- and the taint
query runs for 23 seconds. So the gap is the middle: the flow crosses
`new MemberOrder( $code )` into a constructor and then
`$this->getMemberOrderByCode( $code )`, both dynamic in PHP. Whether
php2cpg resolves either into a callee edge is UNVERIFIED; two attempts to
query the call graph failed on script errors and were abandoned rather than
guessed at. Recorded as 5.7, with the answer required before anything is
concluded about the fact tables.
Three engine defects had to be fixed before the scan could produce even
this result, and each had previously reported the plugin clean for a
different reason:
1577 regions, 0 scanned joern-parse failed where php2cpg succeeded
500 scanned, taint 0.05s a 1.49MB batch over Linux's 128KB argv cap
500 scanned, taint 21.7s the entry point was not recognised as one
The middle one is the one to remember. Every honesty mechanism in this
codebase labels a verdict -- suspicion, regions_unjudged, regions_unasked,
stated family limits -- and there no verdict was produced to label, so a
vulnerable plugin was reported clean with every counter agreeing.
What is established: the WordPress engine path works end to end. What is
not: detection on a real plugin. One checkout that does not find its CVE is
not yet evidence either way about the facts, and 5.7 has to answer whether
the call graph can carry the flow at all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.8. VAmPI's last two false positives were register_user
and login_user, both declared open by its OpenAPI spec: the arbiter was
reporting a missing guard on endpoints whose contract says there is
nothing to miss.
**The fix keys on provenance, not on the value, and that distinction is
the whole task.** The mapper says "public" in two very different
situations. An OpenAPI operation with no `security` block is DECLARED
open, because the spec is a complete contract. A Flask handler with no
`@login_required` is "public" only because nothing was found -- which is
precisely the bug this family exists to report. Suppressing on the value
alone would have silenced the detector on its primary case, so
EntryPointRecord now carries `access_declared` and only a declaration
removes the obligation.
WordPress supplied the second source of declared access on the same day --
wp_ajax_nopriv_ for a logged-out handler, wp_ajax_ for a logged-in one,
permission_callback => __return_true for an open REST route. A rule resting
on one framework's convention would be a special case; two make it a shape.
vampi entailed 6 -> 4
access-control precision 2 of 4 -> 2 of 2
known vulnerabilities 3 of 3 still entailed
access_control offered to 6 of 14 handlers, exactly the six the spec
declares authenticated
**And the guard that decides whether it is a real improvement:** the
access-control pair corpus, which nobody tuned this against, is unmoved --
pair_correct 3 and leak 1 across all nine, pair_correct 2 and leak 0.0%
across the seven from projects unrelated to VAmPI, identical to task 2.6.
The precision was not bought by silencing the family. Pair excerpts carry
no route manifest and no hook, so nothing in them is declared open and the
rule correctly does not fire.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.17, completing Req 5.5.
Task 2.11 made static pattern matches proportionate and left the model
layer's own suspicions alone, reasoning that a proposer had reasoned about
them. That does not survive inspection. A model suspicion is what the
enumerator proposed at a site the graph could NOT decide, affirmed by a
judge with no coverage there -- and its own rationale says exactly that:
"the model resolved no flow here; its sink table may not cover this API".
It is a pattern match with an opinion attached, and it reads like a
reasoned claim only because a model produced it.
vampi 60 findings, 61 sections -> 26, report 1085 lines -> 446
libpng 198 findings, 39 sections -> 30
**What it costs, which is the number that decides whether the rule is
right.** For each known vulnerability the recipe declares, take every
finding row at its location and ask whether any survives the rule. All
three of vampi's -- the SQL injection and both documented BOLAs -- are
still found at model_entailed and all three survive. Nothing known is
hidden. Three vulnerabilities in one repository is also a statement about
how small the evidence base is, and the artifact says so.
The obligations checker sits at `suspicion` too and is NOT grouped: its
output comes from a structural analysis, an obligated operation with no
discharging guard, rather than from a guess. An earlier attempt at this
rule filtered on rung alone and would have collapsed it; six tests failed
and they were right to.
"Pattern matches" is now "Proposed, not corroborated", since it holds two
shapes. Model rows group by FAMILY rather than by rule, because their rule
is always `model`.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.5 recorded that php2cpg emits a CPG containing only one PHP file --
the last alphabetically -- and called it a blocker for the WordPress
target. Rerun with a native php-cli, as asked, and it is not true.
php 8.3.6, no shim, same joern 4.0.623, same two files:
files: plugin.php, vuln.php
methods: mp_fetch_report, mp_save_report, mp_render_box, mp_helper,
get_user, safe_user, show, run
Both files, all eight methods, and no PHP-Parser banner warning at all --
the banner that looked like the thread to pull was a symptom of the same
thing and does not occur natively.
The cause was the workaround. This machine has no PHP interpreter, so the
shim ran each `php` invocation in its own container; php2cpg drives
PHP-Parser once per file, and that per-invocation isolation kept only one
file's JSON.
**The lesson is worth more than the fix.** Two independent reproductions
agreed, and both were measured through the same broken instrument. A
consistent result is not a correct one when every measurement shares an
apparatus -- the same trap as a corpus that cannot see the defect it is
scoring, which is this session's other recurring finding.
And with the instrument fixed, 5.4 is demonstrably done. A two-file PHP
tree yields 10 function-scoped regions and 7 entailed findings in 15
seconds: SQL injection in both WordPress handlers, XSS in the shortcode
renderer, and all three of the original probe's. safe_user is correctly
absent, its mysqli_real_escape_string dropping it to corroborated. The
wp_ajax_nopriv_ handler ranks 1.0 and the plain helper 0.3.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.4. Both halves, and the second found a frontend blocker.
**One verdict per region was right for a pair and wrong for a file.** A
pair is one labelled function with one bug, so one answer is correct. A
region spanning a PHP file is not: the probe's SQL injection on line 5 and
command injection on line 13 produced ONE injection verdict, `system` won
it on flow length, and the SQL injection went unreported by construction
rather than by any failure of the analysis.
`taint.verdicts()` now returns one verdict per distinct sink site,
strongest first, and its first element is exactly what `verdict()`
returns -- so the pair path and every committed measurement resting on it
are unchanged, and the gates are byte-identical. Both entry points share
`_usable_flows` so neither can drift from the other.
php probe 2 of 3 -> 3 of 3
vampi 6 entailed, unchanged, now with lines on access-control
findings from task 2.7 (users.py:187, books.py:51 -- the two
documented BOLAs at their exact lines)
libpng 1 -> 2 entailed; the second was hidden by the collapse.
Neither is validated and both are in contrib/.
**WordPress registers handlers rather than decorating them, and the hook
name is the declaration.** `wp_ajax_nopriv_*` is WordPress saying
logged-out callers reach this, so it is public; `wp_ajax_*` fires only for
logged-in ones; `permission_callback => __return_true` declares a REST
route open. That is the declared-provenance signal task 2.8 needs,
arriving from a second framework. Plus every top-level PHP function as its
own region, without which a file is one region however many functions it
holds: 1 file-level region became 10 function-scoped ones.
**And a blocker, recorded as 5.5.** php2cpg emits a CPG containing only
ONE PHP file, the last alphabetically. Reproduced twice on independent
samples; both files parse cleanly under `php -l` and under PHP-Parser
directly, and joern-parse reports success either way. php2cpg logs "Found
warning in PHP-Parser JSON output" naming the survivor, and that warning is
PHP-Parser's own `====> File X:` banner appearing in what php2cpg expects
to be pure JSON.
Not isolated yet: this machine has no PHP interpreter, so php2cpg was
driven through a container shim. The banner comes from PHP-Parser rather
than the shim, which makes a shim-only cause unlikely but not excluded, and
rerunning with the native php-cli in the shipped image is the first step.
It blocks WordPress outright -- a plugin is many files, and a scan that
silently covers one of them and reports success is the quiet failure Req
5.6 was approved for.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 5.1, the first step toward the v0.1 WordPress target.
Written against a PHP CPG rather than against PHP source, because the two
do not look alike. Verified with joern-parse 4.0.623 / php2cpg on a probe:
$_GET["name"] -> call `<operator>.indexAccess`, CODE $_GET["name"]
mysqli_query($conn, $sql) -> call NAMED mysqli_query
echo $_GET["msg"] -> call NAMED echo
So sources match on code and sinks match on name, and every entry the
probe did not exercise is marked unverified in the file -- `include` and
`require` among them, which is PHP's most characteristic path bug and the
first thing to check on a real checkout.
Five families derive (injection, output_encoding, path, deserialization,
untrusted_destination) with no sink that is its own sanitizer -- asserted
by a test, because that bug class has now appeared three times in three
matchers. `php` is added to the region model's languages, without which
the facts would have reached nothing.
The arbiter decides 4 of 4 on the probe: the SQL concat entails, its
`mysqli_real_escape_string` twin drops to corroborated, `echo` XSS entails
and `system` command injection entails.
**A whole scan finds 2 of those 3, and the gap is the finding.** PHP files
yield FILE-LEVEL regions because no PHP entry points are mapped, and the
taint arbiter returns one verdict per region per family -- so a file-level
region reports at most one injection finding however many it holds.
`system` won on flow length and the SQL injection went unreported. That is
the mirror image of task 2.9: that one is many regions collapsing to one
site, this is many sites collapsing to one verdict, and it bites hardest
exactly where PHP needs it not to, since a WordPress file is large and has
no decorator for the mapper to find.
Recorded as task 5.4 with WordPress's registration model (add_action,
add_filter, admin-ajax, REST routes), which the mapper already handles in
principle for other frameworks.
A 13-line probe is not a WordPress result. What is established is that the
seam works end to end for PHP; everything about scale and coverage is
unmeasured, and there are still zero PHP pairs in a corpus of 263 python,
351 c_cpp, 47 javascript and 49 java.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting
that `memcpy` and `strcpy` are PRESENT in a library that uses them
correctly throughout. That is the noise developers have learned to skip in
every other tool, and shipping it costs the findings beside it their
credibility.
**Filtering on rung was the obvious rule and it was wrong.** The
obligations checker emits at `suspicion` too, so hiding that rung would
have hidden an entire detector -- six tests failed and they were right to.
The discriminator is evidence_level: on libpng 163 of 198 rows are
`static_corroboration`, on vampi 3 of 60 are, and the model layer and the
obligations checker carry `suspicion` there instead. That is exactly the
distinction the ladder exists to make -- an undecided CLAIM about a site
is not an observation that an API exists.
So a pattern-only rule is grouped once it has more than three sites, and
keeps its full sections under that. One `python-flask-debug` is a finding a
contributor can act on and must not be summarised away; fifty-four
`c-unsafe-copy` are a property of the codebase.
libpng 198 findings, 198 sections -> 39, 161 grouped into three lines
vampi 60 findings, 61 sections -> 61, nothing grouped
Counted, never dropped: findings.json stays complete because it is the
record a baseline is compared against, and the grouped section says the
model reached none of these and that its silence means no fact table
covered the site rather than that the site is safe.
What it does not fix, recorded rather than glossed: vampi still emits 61
sections for a 520-line application, 51 of them reasoned claims this rule
does not touch. It fixes pattern noise, not volume.
Also records what group 2 established about the three corpus asks, so none
is started on a wrong assumption. Joern ships php2cpg but our fact tables
have zero PHP families, so WordPress would parse and produce nothing --
fact tables first, corpus second. An older vulnerable libpng will not help
for a reason no version can fix. Vibe-code repositories are the best fit
and the cheapest route to a second true-positive checkout, because Python
and JavaScript are the two languages whose fact tables are complete.
And Group 8, for per-repository state a developer owns: a committed
dismissal file the tool reads and never writes, dismissals as a split
evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION --
the judge and residual prompts -- never to the fact tables. That boundary
is the whole design: a prompt is how we ask, a fact table is what we
claim, and the closed-loop leak measured on 2026-09-05 is why the
distinction is not negotiable.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.12. c/injection and c/memory carried zero sanitizers,
so no C flow could ever be discharged: every `argv -> strcpy` entailed
however carefully it was bounded.
Which sinks can be discharged this way is read from the sink fact's own
CWE rather than declared: CWE-121 is a stack buffer overflow, so
gets/sprintf/strcat/strcpy qualify. It is the only overflow CWE in the
fact tables and it is C-only, so no Python or JavaScript sink is touched
and the pair corpus is unaffected -- gates byte-identical.
Not a sanitizer list, because `strlen` appears in the guarded and the
unguarded case alike; the discharge is a guard, not a call.
**The first implementation required the bound to DOMINATE the sink, and it
found nothing.** libpng guards `strcpy(outname+len, ".png")` with
`(len = strlen(inname)) > 250`, but that check sits on an else-if chain
whose sibling branch skips it entirely, and the protection reaches the
sink through an `error` flag: `++error` in the guarded branch, `if
(!error)` around the copy. Joern's dominator set for that sink has 89
calls and correctly excludes the bound; `controlledBy` returns none
either. Verified against the CPG before the rule was changed rather than
after it failed.
So the bound is searched in the method, and the consequence is
corroborated rather than silence. That is the honest rung: a bound on this
variable means the graph has NOT shown the copy is unguarded, and it has
not shown it is guarded either -- off-by-one being how bounds fail. The
judge is asked. Proving the flag-mediated case needs path-sensitive
reasoning this arbiter does not do.
libpng entailed 3 -> 1
The survivor is also a false positive: `strcpy(output + len, ".icc")` in
iccfrompng.c, where `output = malloc(len + 5)` makes it fit exactly. That
is a different discharge shape -- allocated to size, not bounds-checked --
and it is recorded as 2.13 rather than chased. Both shapes were found on
one repository, and extending a rule until that repository is clean is the
overfitting Req 8 exists to prevent; 2.13 requires a second C checkout
with a genuine CWE-121 so the change can be measured for what it silences
as well as what it clears.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.10. Every one of libpng's twenty-five entry-point
regions was a `main()` -- twenty-three under contrib/, plus example.c and
pngtest.c -- and those regions carried the fifty interprocedural taint
requests that dominated a 300s query phase. A library has no entry points,
so entry-point-driven analysis had nothing to anchor to except its own
example programs.
The task said to derive this rather than hardcode `contrib/`, and a project
already declares it. Makefile.am names installed targets in
`lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES`
names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own
word for "not installed" -- exactly the distinction wanted. Taking every
`_SOURCES` would have called libpng's whole test suite shipped code, since
contrib/libtests declares its sources the same way the library does.
interprocedural regions 25 -> 1
taint query 300.1s -> 125.0s
total query 311.9s -> 138.5s
regions 121 -> 121, unchanged by design
The surviving interprocedural region is contrib/tools/pngfix.c:main, which
libpng genuinely installs.
Two rules keep it honest. **Silence is not exclusion**: a project with no
build file this can read gets None back and nothing changes, so vampi's
numbers are untouched. **Not shipped is not unscanned**: those regions are
still arbitrated function-locally, because a contributor editing a test
must still get an answer -- what they lose is the expensive treatment and
first claim on the budget.
Entailed findings went 0 -> 3, and all three are false positives at
contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe:
outname is char[256] and line 325 guards with
`if ((len = strlen(inname)) > 250)`. They were not introduced so much as
uncovered -- a different callDepth meant a different sink set, and the 26
fgets false positives were masking them.
The structural cause is recorded as 2.12: c/injection and c/memory carry
ZERO sanitizers, so no C flow can ever be discharged and every
`argv -> strcpy` entails however carefully it is bounded. The discharge
here is not a call but a guard that dominates the sink, which is precisely
what dominance.sc already does for access control and is wired to that one
family.
Also adds Group 7, five tasks for stopping the same false positive twice.
Two of today's three false-positive classes were one bug in three
matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in
`fgets` -- each fixed locally and each next one surviving because nobody
carried the fix across. A model may group counter-examples by cause and
draft an edit; only a split-respecting measurement may adopt one, because
the predecessor's improve lever admitted shapes taught by the holdout pairs
they then recovered and turned a -0.059 Youden into +0.059.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform.
89,082 lines of C across 93 files, under a 2GB heap cap:
CPG build 16.1s (25s cold)
Joern queries 250.0s
arbitration 122.0s (105 calls, $0.0122)
whole scan 446s
peak RSS 911MB
regions 121, all arbitrated, 32 unasked
CVE-2025-66293 out_of_scope, as the recipe declares
Memory is not the constraint a fleet runner would hit: 911MB leaves room
for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen
while building this was concurrent JVMs on one desktop, not repository
size.
**Server mode: no.** The query phase splits taint 243.3s / config 10.7s
over two invocations, and fixed overhead measured on the same CPG is ~13s
per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL
evaluation: a warm server removes 6% of the scan and none of the part that
costs. The driver now records query seconds per kind, because a single
total cannot tell startup from evaluation and the two answer this
oppositely.
The small repository pointed the other way. vampi's 34.3s over three kinds
is roughly three startups and almost no evaluation, which made a warm JVM
look worth having; the ratio inverts with size. That inversion is why the
task was sequenced after a real repository rather than decided on a
520-line file.
On whether one whole-repository CPG is the only shape for a horizontal
fleet: no, and it is also not where the pressure is. The build
parallelises and costs 16s; the query does not shard and costs 250s, so
splitting a repository across runners divides the cheap phase -- and after
2.4 a shard boundary hides exactly the cross-module flows that matter.
Scoping to a contributor's diff cuts the expensive phase instead. 911MB
for 89k lines extrapolates to roughly 5GB at a million lines, which would
exceed a standard runner, so the ceiling is real and further out than this
checkout can locate.
Two defects recorded rather than fixed, both invisible at vampi scale:
2.9, all 26 entailed findings are the same site reported 26 times, because
regions that share a sink each emit their own; and 2.10, 22 of the 30
highest-ranked regions are libpng's `contrib/` example programs rather than
the library.
Also ignores Joern's `workspace/`, which a maintainer querying a CPG by
hand from the repository root will create.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.6. Access control entailed 7 of 9 VAmPI handlers,
including three that do authorize and three the OpenAPI spec declares
public. Five separate defects, each invisible to a corpus of one-function
excerpts and each found by asking the query what it actually matched.
**A trailing segment is not a match.** `query.get` was matched by its last
segment, so every `.get(` in the repository became a protected database
operation -- `request.headers.get('Authorization')` and
`request_data.get('username')` included. Four of the seven false positives
came from this, and so did the apparent detection of the known BOLA: it was
entailed because fetching the Authorization header had no dominating guard,
not because of the defect. A qualified token now requires its qualifier.
**Configuration is not authorization.** dominance_specs took every
obligation fact. `security_setting` operations are discharged by a VALUE,
which config_specs already claims them for; pooled in here they made
`app.run(...)` an operation requiring an authorization guard and put
thirteen tokens in BOTH lists -- an obligation that is its own discharge,
the same shape as the taint sink that used to sanitize itself.
**Input validation is not authorization.** `jsonschema.validate` was
counted as a guard, so handlers with no authorization check at all read as
guarded. No operation in the table requires validated_input.
**A guard discharges the obligation it is a guard of.** The largest one.
`orm-read` requires an identity_constraint or an ownership_check;
`token_validator` is a path_guard. Flattened into one list, authenticating
the caller discharged an object-level obligation -- a handler that
authenticates and then looks the record up by a PATH parameter read as
fully guarded. The facts state that relation and the spec was discarding
it; it is now carried and the engine asks per operation.
**A word boundary needs a word.** Found while fixing the above. Guard
matching wrapped every token in `(?<![A-Za-z0-9_])...(?![A-Za-z0-9_])`.
The identity token `['sub']` begins with `[`, and the character before it
in `resp['sub']` is `p` -- so the assertion could never hold. Every
identity token starting with punctuation was unmatchable, and the identity
constraint that separates a correct lookup from an IDOR was invisible.
Also: dominance rows were repository-wide, so the model's unguarded
`delete_user` was attributed to the API handler of the same name. Rows are
scoped to the region's file, which is what the rung's own justification
says -- the evidence is a file contradicting itself.
Measured against VAmPI's OpenAPI `security` blocks, which declare per
operation whether auth is required, and agree with the code:
vampi precision 28.6% (2/7) -> 50.0% (2/4)
known found 3 of 3, all model_entailed
And on the labelled access-control pairs, which nobody tuned this against:
all 9 pairs pair_correct 3 -> 3, leak 44.4% -> 11.1%
7 non-VAmPI pair_correct 2 -> 2, leak 42.9% -> 0.0%
Every detection lost was on a pair whose FIXED side was flagged too, which
scores nothing. No correct detection was traded for the precision.
get_by_title was flagged before it was in the recipe: its code carries the
comment "# Broken Object Level Authorization" and the README lists it, so a
real finding was being scored as a false positive. It is now the recipe's
third known vulnerability.
Both remaining false positives are endpoints the spec declares public.
Recorded as 2.8, with the reason the mapper's access_level cannot simply be
trusted: for decorator frameworks "public" means "no decorator found",
which is the bug itself.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.5. VAmPI's `host='0.0.0.0'` is now reported exactly
once, at app.py:17:<module>. Before task 2.2 it was reported eight times
in eight files that do not contain it; after 2.2 it was reported nowhere.
Two causes, one shape: nothing could ask about module-level code.
**The main-guard marker is not a function.** `__main__` is what the mapper
emits for `if __name__ == '__main__':`. Sent to the engine as a function
name it matched nothing, because a CPG calls a file's top-level code
`<module>` -- verified against the VAmPI CPG rather than assumed. The
region model translates the marker, which is the one place that turns what
the mapper meant into what the scanner asks.
**A function-scoped file had no region for its module body.** Settings,
route registration and globals live in no function, and every region's
query filters by function name. A file-level region beside the handlers
would report each handler's findings a second time, which is why the
earlier rule dropped it as "a second full-file pass". A module-scope region
PARTITIONS the file instead of overlapping it. That only became cheap after
batching: before it, a region was a JVM.
Two things this needed to be correct rather than merely present:
- A module method spans its whole file, so the taint query's line-range
containment made `<module>` the parent of every function in it. Under
module scope, membership is direct -- otherwise the module region reports
every handler's sinks again, which is the duplication the old rule was
avoiding.
- A module body is not a handler. The marker-derived region inherited the
handler families and entailed a missing authorization check on the main
guard: a finding with no possible remediation. Caught by the first
repository run that had the region, and now covered by a test.
bind address 8 wrong files -> 0 -> 1, at app.py:17:<module>
regions 22 -> 25
entailed 8 -> 9
query time 37.4s -> 41.3s
peak RSS 827MB -> 823MB
cost $0.0067 -> $0.0068
Three more regions for four seconds and no more model calls: the budget is
a total and was already spent, so regions_unasked rose 17 -> 20 -- those
regions got the free pass and are reported as unasked, not as clean.
`<module>` is Joern's Python name for a module body; JavaScript uses
`:program`, so this is verified for Python only.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.4, closing the no-go task 2.2 recorded.
VAmPI's SQL injection is a request parameter reaching `get_by_username`
in api_views/users.py, passed to `User.get_user` in models/user_model.py,
interpolated into a query string and executed there. Source and sink are
in different functions and different modules. Regions were per-function,
so neither region saw both ends and the engine was silent on the most
ordinary shape a real injection bug has.
`reachableByFlows` was never the limitation -- it follows calls already.
The limitation was the QUESTION. Sinks were filtered to the labeled
function's own body, so a crossing flow had no sink to end at.
Three changes, each with a reason it is sound rather than convenient:
**An entry point is the trust boundary.** parameter_sources was off for
repository scans because a repository's parameters are not all attacker
input. An entry point's are, by definition -- that is what makes it an
entry point. The driver turns it on only for regions the mapper found and
named, and leaves it off for walker fallbacks and for named functions that
are not entry points.
**Sinks may sit below the handler.** taint.sc admits sinks in methods
reachable from the labeled method through the call graph, bounded at
ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is
where these bugs live. Unbounded would make the question about most of the
repository rather than about this entry point, and make it unaffordable.
**The finding names the sink, not the asker.** A crossing flow ends in
another file and another function, so Verdict carries the evidence's own
`path:line:function` and the site uses it. Reporting
`models/user_model.py:73:get_by_username` would name a place that does not
exist: line 73 is inside get_user.
Also: labeled methods are now resolved by name AND file. `get_user` exists
in more than one module of a real repository, which a corpus of one-file
excerpts could never surface.
Measured on the same checkout and budget as 2.2:
VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user
known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now
entailed 7 -> 8
query time 40.9s -> 37.4s
peak RSS 869MB -> 827MB
cost $0.0067, unchanged
Asking the bigger question cost nothing measurable here: file scoping and
call-graph following pull in opposite directions and roughly cancel at this
size. That is one 520-line repository, and VAmPI is now the repository this
change was developed against -- the exact overfitting shape Req 8 exists to
catch. The depth bound and the entry-point rule mean nothing general until
a checkout nobody tuned against says so.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.
The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.
**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.
**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.
**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.
**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.
Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.
Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The pair corpus scores excerpts: a vulnerable function and its fixed twin,
handed to the engine with the answer already narrowed down. Nothing in it
exercises a repository -- region ranking, the budget, one CPG over
thousands of files, or whether a verdict that holds on fifteen lines still
holds when the source, the sink and the guard live in three modules.
benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char
commit SHA, licence, and every known vulnerability with its file, function
and line. `ousast repos` lists them; `--fetch` materialises one; CI does
neither, and conftest clears the switch and redirects the cache so a
developer with warm checkouts cannot read them by accident.
Three rules the loader enforces rather than documents:
- A branch or tag is rejected. Either would make the same recipe mean
different code next month while the baseline it produced kept the old
numbers.
- `in_scope` must be stated per vulnerability, never defaulted. It decides
whether "not found" is a miss or a non-question, which is the most
important thing about the result.
- A recipe that declares `detection` must name something in scope to
detect.
`ousast repos` does not trust a recipe either: against a materialised
checkout it checks HEAD against the pin, that the licence file is there,
and that each declared function appears above its declared line. That
earned its place immediately -- the first libpng recipe named LICENSE.md,
which is the filename at HEAD today and not at the pinned commit, where it
is still LICENSE.
Two checkouts, verified against the real repositories:
- vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are
already in the pair corpus, including the update_password miss diagnosed
earlier, so a whole-repository run says directly whether a verdict
survives the move off the excerpt. Its routes register through an
OpenAPI operationId, so it exercises registration-based entry-point
mapping rather than the decorator path.
- libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at
the parent of the commit that added the missing clamp. Its CVE is marked
out of scope: an out-of-bounds read is not a family this engine
arbitrates, and recording "not found" against it as a miss is the
easiest way to make a measurement lie.
Nothing is vendored; the recipe records the licence so an artifact can say
what it ran against.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Pair scoring could rely on a labelled function; a contributor cannot supply one
-- which code is risky is the question they are asking. Regions now come from
entry points, with a file-level fallback that says it is one, and each carries
only the families its shape and language admit. Ranking uses what
EntryPointRecord already decided (access_level, trust_boundary) rather than a
second opinion computed from the same evidence.
A family is offered only where its arbiter could answer: access_control needs a
handler, so a library file with no entry point is not asked about obligations it
has no siblings to be inconsistent with.
The driver makes the two costs that were free at pair scale explicit. ONE CPG
per repository, reused across regions -- a build is 5-30s on a forty-line
excerpt and per-region builds would make this unusable. And a budget that is a
TOTAL for the run rather than the enumerator's per-region cap of eight, spent
highest-rank-first, with whatever it does not reach counted into
regions_unjudged. A tool that silently analyses a tenth of a repository and
reports a clean result is worse than one that says what it skipped.
Two flaws the tests caught, both mine and both about ownership: the budget
counted calls by reading a client's own usage log, so it silently never tripped
for a client that keeps none -- the driver now counts its own spend through a
wrapper; and the driver trusted its caller to have sorted, when the order the
budget is spent in belongs to whoever holds the budget.
The third was worse and only surfaced because a test failed for the right
reason: the driver passed candidates=() and never enumerated, so a repository
scan would have reported only what the arbiters establish unaided and never
populated the suspicion band -- which supplied 15 of 26 findings in the pair
measurement. It enumerates now.
Sibling audit (Req 8.5) done before writing: family routing lives in exactly one
production place, pipeline._arbitrate, dispatching on spec type. regions_for
returns family names and the driver resolves specs, so that stays true.
750 passing, mypy clean over 96 files, gates byte-identical, zero orphans.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.
Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.
Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 90.0%
leak 12.3% 6.7%
detection -- 96.7%
cost ~$0.69 $0.0068
27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).
One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.
The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 86.7%
leak 12.3% 6.7%
detection -- 93.3%
cost ~$0.69 $0.0067
Ten points of recall over the hunter, half the leak, a fifth of the model calls,
~100x cheaper. 26 of 30 pairs correct.
Four runs, and each step came from a measurement contradicting an assumption --
twice one I had stated:
1 73.3% the absence and config families had no pipeline path at all
2 73.3% routing fixed and the result IDENTICAL, which was the clue: the
arbiter was reached and its verdict thrown away, because every
corroboration was put to the judge as a taint question
3 80.0% residual question phrased for the arbiter that raised it
4 86.7% aiohttp Application(debug=True) and RedisStorage(httponly=False)
added to the settings table
The division of labour keeps moving the right way: model_entailed 9 -> 11 while
suspicion held at 15, so both findings gained in run 4 came from the MODEL, not
the LLM, and cost fell from $0.0126 to $0.0067 as work moved onto the free
deterministic path. That is the payoff curve for fact-table coverage, measured
rather than argued: closing a gap moves findings out of the LLM's column into
the model's, improving precision and cost at once.
Recorded corrections and limits, because the numbers are only worth what the
caveats are. The three silent config pairs were called corpus defects on a
similarity threshold borrowed from the OWASP minimal-fix analysis; it does not
apply to a slice that deliberately pairs a vulnerable function with a different
safe one, where 36% and 34% pairs both score pair_correct. Two were a settings
gap and now split. And permissive_values is a flat literal list with no keyword
association, holding both True and False -- debug=True is dangerous where
httponly=True is safe, and the model cannot tell them apart. It did not bite
here, and it is a real limit of the fact schema.
Four remaining misses, each with a named cause. Four runs cost about $0.04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 80.0%
leak 12.3% 6.7%
cost ~$0.69 $0.0076
determinism none entailed band deterministic
Higher recall, half the leak, a fifth of the model calls, ~90x cheaper.
Three runs, and the middle one earned its keep by changing nothing. Run 2 fixed
family routing and returned numbers IDENTICAL to run 1 -- which was the clue
that the dominance arbiter was being reached and its verdict thrown away. Every
corroboration was put to the judge as a taint question ("a sanitizer appears on
the path -- does it suffice?"), meaningless for a missing guard, so the judge
said no and the finding was dropped. Run 3 phrases the residual question for
the arbiter that raised it and converts 2 of 3 access_control pairs from silent
to pair-correct, arriving at model_corroborated -- a rung that existed but had
never reached a finding.
This settles a question open since the start of the session, and reverses an
earlier answer of mine. "Detection has regressed" was measured with the
proposer disconnected: an arbiter compared against a detector, before four
wiring bugs were found.
Division of labour: 15 suspicion, 9 model_entailed, 2 model_corroborated. The
model arbitrates 11 of 26 findings, and every conversion from suspicion to a
model rung removes a leak and a model call at once -- the payoff curve for sink
coverage and interprocedural resolution, now measured rather than assumed.
Still silent: 3 config_secrets (the settings table is Flask/FastAPI-shaped
while the pairs are aiohttp) and 1 access_control still to diagnose.
Three full runs cost about $0.03 in total.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The first like-for-like comparison with the LLM hunter, on the same 30 vibe-py
pairs and the same metric.
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 73.3%
leak 12.3% 6.7%
cost ~$0.69 $0.0126
determinism none entailed band deterministic
Half the leak, 55x cheaper, a fifth of the model calls, one pair short of the
hunter's recall (22 against 23 of 30). And understated: all six both_silent
pairs are access_control and config_secrets, which had no pipeline path to
their arbiters when this ran. A corrected run supersedes those rows.
The rung split says what is doing the work: 15 suspicion to 9 model_entailed on
the vulnerable side. The LLM supplies most of the recall; the model supplies
the precision, halving the leak by dropping claims it could contradict. That is
the division of labour the architecture was designed around and the first
evidence it pays.
It also revises an earlier conclusion of mine. "Detection has regressed" was
measured with the proposer disconnected -- an arbiter compared against a
detector. With the enumerator wired in, the comparison reverses on precision
and cost while holding recall.
And it makes the strategy concrete: the model arbitrates about a third of
findings today, and every suspicion converted to model_entailed is leak removed
for free and one fewer model call. Sink coverage and interprocedural resolution
now have a measured payoff rather than a hoped-for one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Everything before this built an ARBITER: something that judges a flow taint
reachability had already found unaided. Nothing proposed to it. That is why its
recall was never comparable to a detector's, and why comparing 41.3% entailment
against the LLM hunter's 76.7% pair-correct was comparing two different kinds
of thing. model/judge.py and enumerate_candidates were called by no production
code at all; every number so far came from a measurement harness.
The wiring turns on a distinction judge() alone conflated:
the model CONTRADICTS it has coverage at this site and says no -- a
sanitizer on the path, or a declared-safe sink shape.
The claim is dropped (Req 8.2).
the model CANNOT DECIDE no modelled sink or source reaches the site at all.
The model is SILENT, not negative, so the claim is
reported at `suspicion` (Req 8.4).
Treating silence as refutation is the tempting mistake and would be wrong here:
23 of the 39 misses in the entailment ceiling were sinks simply absent from the
fact tables, so "no flow" usually means the model cannot see it rather than
that nothing is there. The rung carries that uncertainty on the finding's face
instead of a confidence score nobody can act on.
An entailed site still costs zero model calls. The candidate budget is bounded
so a dense region cannot become an unbounded spend.
Smoke-tested end to end against a live endpoint: 2 pairs, detection 100%,
$0.0024. The full vibe-py measurement -- the first like-for-like comparison
with the hunter's 76.7% pair-correct and 12.3% leak -- is running.
727 passing, mypy/ruff clean, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
output_encoding had exactly ONE pair in the whole corpus. It now has 41.
238 pairs were harvested from OWASP BenchmarkJava (2740 generated servlets,
GPL-2.0, with a ground-truth CSV) by matching each really-vulnerable test to
the safe test of the same category it most resembles -- adjacent tests come
from one template, so they differ in the sink or the sanitiser rather than
being unrelated programs. Median similarity 0.91, against the 64% of the older
`sast` OWASP entries that the group-4 stratification flagged as different
programs.
Only 40 are vendored. One representative of each of the seven categories was
run through the model BEFORE vendoring, and only xss distinguished: its fix
adds ESAPI encoding, which the Java sanitizer facts recognise, so the
vulnerable side entails and the fixed side drops to corroborated. cmdi, sqli,
ldapi and xpathi put the sink in a helper class outside the servlet and
resolved no flow at all; pathtraver found the flow but its "safe" twin carries
no distinguishing call, so text similarity had matched a twin that is not
semantically the fix -- a subtler form of the very defect the harvest was
designed to avoid. Those 198 pairs are deliberately left out: adding pairs the
model cannot read would inflate the corpus while making every number less
interpretable.
Evidence for the vendored slice is 3 of 3 distinguished, which is a small
sample and is stated as such in the catalog header rather than implied to be
more.
The slice is marked SYNTHETIC in its header: it measures breadth across a
weakness class and must never stand in for real-world performance. The corpus
already shows a +16% overfitting gap, and a large generated slice blended into
a headline would widen it invisibly.
Two things recorded rather than hidden: BenchmarkJava's LICENSE is GPL-2.0
while the older sast entries declare GPL-3.0 for OWASP, which looks like an
error and is not copied; and OWASP-Benchmark/BenchmarkJava already straddled
the train/holdout split before these pairs (owasp-java-cmdi train,
owasp-java-sqli holdout), so the new all-holdout pairs enlarge an existing
straddle rather than create one.
The two vendored-count guards did their job -- a corpus change could not land
silently -- and are updated as the acknowledgement. Hygiene artifact
regenerated: 277 -> 317 vendored pairs.
720 passing, mypy/ruff clean, gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Java shipped with injection, deserialization and crypto sinks only, so the path
and output_encoding families had nothing routing to them by CWE and any Java
pair in either was unreadable -- the gap hutool-cve-2018-17297 reported as "no
path spec for java". Adds the JDK filesystem entry points, the servlet response
writers, the outbound/redirect APIs, and the ESAPI / canonical-path sanitizers
the OWASP suite itself uses. Documented APIs, provenance per entry.
Post-closure-fix ceiling: 41.3% entailed (38/92) and 26.1% distinguished, up
from 39.1% / 23.9%. path went 0/8 to 1/8 with its first distinguished pair;
injection 29->31 entailed and 18->19 distinguished. Two points -- real, and the
mechanism is understood, but the remaining gaps are data rather than machinery.
Carrying a correction: the measurement harness routed config_secrets through
TAINT rather than the constant-abstraction arm, so its 0/10 in this artifact
measures the wrong arbiter and did so in both full runs. The harness is fixed;
that family's real number is the earlier targeted re-measure, and this
artifact's config row should not be read.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The rung this project deliberately does not build. Clearwing already runs
sandbox lifecycles, container pools, sanitizer images, PoC replay and stability
classification, and the corrective brief concluded -- from this project's own
defect history -- that reimplementing that machinery is where we go wrong. So
model/execution.py decides eligibility, calls an adopted tool, and records what
happened. A test asserts structurally that it imports no container or
subprocess machinery and defines no build/run/compile/patch function, because
the failure mode to prevent is this file quietly growing a sandbox.
Eligibility is narrow in both directions: only a family a static model cannot
arbitrate (memory safety in C, where the bug turns on runtime layout -- Req
10.3 forbids any web/logic family depending on this tier, and each now has
taint, dominance or constant abstraction instead), and only a finding the model
left undecided. The tier escalates uncertainty; it never re-checks a verdict.
One deliberate asymmetry: a failure to reproduce is NOT a refutation. A sandbox
that does not trigger a bug has not shown the bug is absent, only that this
attempt did not trigger it, so the candidate stays where the model left it.
Reading a failed reproduction as a clean bill of health is the same error class
as reading an absent engine as "no vulnerability".
With no adopted tool present -- the normal case -- the rung is never reached and
clearwing_unavailable is recorded.
All 16 tasks of model-grounded-detection are now done. 720 passing, mypy/ruff
clean, gates byte-identical, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
5.1 inverts what the corpus is for. Previously a pair the detector missed was a
DETECTOR failure, to be averaged away over K runs and chased with a prompt
change. The arbiter is deterministic: it will miss that pair the same way every
time, and the only thing that moves it is better modelling. So a pair the model
cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every
gap carries one even when the caller supplies none -- a bare gap teaches
nothing.
`covered` means the model reached a strictly higher rung on the vulnerable side
than on its fix. Flagging both sides equally is not coverage: it distinguishes
nothing, which is exactly what group 2 measured taint reachability doing on
every parameterised SQL pair. The harvest seam takes the fix commit and derives
the vulnerable side as its parent, with the fix diff as the oracle -- nothing
hand-labelled.
5.2 keeps three disciplines from the deleted publish layer, each because its
absence caused a real error here:
* per-slice rows always -- `render` RAISES without them. A mixed headline
once hid agent-written code scoring 44% against the development slice's 83%.
* the overfitting gap beside the headline, and ABSENT rather than zero when a
slice is missing: reading an unmeasured slice as "no overfitting" is the
flattering failure, not a neutral one.
* the rung on every finding, so a suspicion cannot be read as a result.
Run end to end against the committed ceiling artifact, the report says things
worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an
overfitting gap of +16%, and gaps grouped by family with their reasons --
injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no
verdict), untrusted_destination 5 (4 entailed on BOTH sides).
713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
A full re-measure with the closure-scoping fix is in flight; these numbers are
the pre-fix baseline and will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching
any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter
that makes no model calls at all.
Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36
distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10,
path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the
class no crash oracle can reach, arbitrated deterministically.
A hypothesis tested and discarded. From the group-2 misses I diagnosed path's
0/8 as a missing req.url source token and said so. The probe -- the same 8
pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real
cause is that the sink sits lexically inside an anonymous callback, so its
enclosing method is the closure (<lambda>0) and the sinkMethod == function
filter drops it; verdict(function="") on the same CPG returns model_entailed.
And the obvious fix for that is wrong. Relaxing the filter raises the number
with a FALSE witness: the flows found in the closure are `res -> readFileSync`
and `this -> ...`, neither of which is the vulnerability, whose real path
crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a
miss into a wrong entailment is worse than the miss -- it is the arbiter
asserting something untrue, the one failure mode this architecture exists to
prevent. The correct fix is two-part and recorded: scope a sink to any method
lexically nested in the labeled function, and scope parameter sources to that
function's own parameters, then re-measure counting only real witnesses.
Every miss is classified and none is architectural: 23 sink absent from the
excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source
unmodelled with the idiom named.
Two honesty items in the artifact. 16 of 92 pairs carry no function label and
are arbitrated over the whole file, so a verdict can credit a different finding
than the labeled one -- owasp-python-hash (CWE-327) is credited for a
permissive set_cookie elsewhere in its file, on both sides. And config_secrets
conflates three abstractions: permissive setting values (implemented), weak
algorithm choice (facts already carry weak_literals, unwired) and hardcoded
secrets (needs a credential-literal check).
694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
A configuration bug has no flow to follow and no guard to dominate.
CORS(app, origins="*") is dangerous because of the value it is SET TO, so the
arbiter evaluates the setting's arguments against the closed permissive set in
the obligation facts -- the third form beside taint reachability and guard
dominance, and the one Req 6.3 names.
ENTAILED a security setting is passed a literal in the permissive set
CORROBORATED the argument is computed, so constant abstraction cannot decide
it -- that residual is the judge's, never silently "safe"
(none) every literal is outside the set, or the call is not a setting
Two Joern node-shape bugs, both found by running it and both of the same class
as group 2's receiver bug -- the graph's indexing is not what the source text
suggests:
* a keyword argument (`origins="*"`) is an assignment-shaped node, so
isLiteral is false on the argument itself and the literal sits in its
subtree; literals are collected from the argument's AST now.
* Joern indexes the receiver at 0, positional arguments from 1, and NAMED
arguments at -1. Settings are configured overwhelmingly by keyword, so
argumentIndexGt(0) -- correct when counting arity for the taint shape test
-- dropped exactly the arguments that matter and every keyword-configured
setting read as "computed".
Live, deterministic, zero model calls:
create_app -> model_entailed CORS(app, origins = "*"): permissive literal "*"
safe_app -> no finding (origins = "https://app.example")
All three arbiter forms now work against a real CPG: taint reachability for the
flow families, guard dominance for absence, constant abstraction for config.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
