mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
ebcc2cf7fd847aa6ff7a5d08b6d9dce340f5ed4f
176
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marius CiepluchandClaude Fable 5.1
|
60d13d7b6e |
feat(model): phase 2 of flow-aware-ranking -- spend the region budget by evidence, weights published
`Evidence.score` orders pairs within a tier by one table of weights (open sinks, cleansed sinks, local / carried / near sources, entry, declared public), capped so volume stops counting, and zero for tier 0 so the weights can rank but never manufacture. `ScanBudget.order_by_evidence` runs the evidence pass over every region and cuts `max_regions` from each region's best (tier, score); `order_by_evidence()` is the one ordering function, used by the scan and by `benchmarks/ranking/position.py --evidence`. Off by default until the position benchmark says where the pinned CVEs land under it. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Fable 5.1
|
88ada8e48c |
feat(ranking): phase 0 -- the baseline, the evidence instrument, and a 7x taint speedup found on the way
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
07df314e12 |
fix(cpg): prove the frontend can read the source before building, and check the graph always
Two permanent guards, and a correction that motivates both.
THE CORRECTION. There is no php2cpg defect. Every PHP build on this machine
ran through a local `php` shim proxying stdio through `docker run`, and
php2cpg depends on the INTERLEAVING of its parser's stderr banners with its
stdout JSON to attribute each document to a file. The proxy scrambles it:
streaming shim drops 72-88 767KB / 969KB / 1.2MB different every run
buffered shim drops 0 5,703 bytes empty, and silent
NATIVE php drops 0 2,872,996 bytes identical x3
The 637-file tree with native php: 0 drops, a 4,329,471-byte CPG, 20.0s,
1.9GB peak -- which is exactly what task 5.10 said before I started
"correcting" it. No intermittent failures, no malformed graph, no dataflow
assertion, no scale limit.
GUARD 1: the interpreter must prove it can read the tree, before anything is
built. Every instrument failure this project has had takes one shape -- the
input was unreadable and the tool reported a plausible ZERO -- and it has cost
four wrong diagnoses in a day:
a php that could not see ~/.cache -> "cannot handle 637 files"
a php whose stdio was proxied -> "drops half its batches"
a container missing a bind mount -> "0 dropped", 5,703-byte graph
None announce themselves: file_exists() is false, the parser writes nothing,
the frontend exits 0, and "no errors" reads exactly like success. Checking the
output cannot separate that from an empty repository. Checking that the input
arrived can, and costs one exec.
GUARD 2: the graph census is taken on EVERY build, not only one the frontend
warned about. A graph holding nothing arrives with no warnings at all --
silence is the symptom that has no symptom -- and the shortfall is now
reported as "the frontend reported no failures but the graph holds N of M
source files".
`cpg_build_failed` carries that detail through to the report instead of being
a bare reason.
Also adds tests/test_cpg_queries_compile.py: the queries are 900 lines of
Scala that nothing compiled. A rowsFor signature change shipped with 850 tests
passing and broke every query at runtime, found by a measurement failing hours
later. Each query is now compiled AND run against a real graph wherever joern
exists, and must return a parseable payload.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
f3f41e2fa9 |
docs: the 637-file wall is the taint query, not memory and not vendored code
Follow-up measurements, because the obvious explanations were all wrong.
Not memory: at 2GB AND 4GB the CPG never finishes loading, with no
OutOfMemoryError either time, both runs stopping in the same place --
Braintree\Util. What costs is Joern's dataflow overlay, recomputed per import.
Not scope, though scope is real and worth fixing on its own terms: 429 of the
637 files are vendored SDKs (Braintree, Stripe, each with its own LICENSE) and
31 are tests. PMPro's own code is 177 files, 62k lines. Auditing a plugin by
scanning the payment SDK it bundles is wrong regardless of cost -- those
findings are not the plugin's bugs -- and it is the same idea shipped.py
already applies to Makefile.am, which PHP has no equivalent of.
Scoping to authored code helps and does not fix it:
regions 4,463 -> 1,324 total 996.6s -> 843.7s findings still 0
600.3s of that 843.7s is the taint query hitting QUERY_TIMEOUT_SECONDS (300s)
on each of two shards. Two timeouts, not work.
So the wall is the per-request cost of the taint query at 2,500 requests. The
next measurement is what the two-stage joins cost at scale -- 1.6x on a slice
(62s -> 97s) and plausibly the dominant term here. Raising the timeout before
knowing that would only buy a slower way to find out.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
c03944f7dd |
docs(benchmarks): the whole-plugin envelope, next to the slice results
The class-by-class table is measured on one- and two-file slices, which invites the reading that the engine scans WordPress plugins. It does not, and the README should not let that stand unqualified. 637 files, 117k lines: the build is affordable (33s, 1.2GB, a 1.6MB CPG) and a CI runner can do it. The scan is not -- 996.6s to decide nothing, with the taint query dying in ForkJoinPool and 500 of 4,463 regions examined. What the run does correctly is refuse to look clean: files_unparsed, query_failed, cpg_sharded and the region cap all reach the reader. A scan that decides nothing and says so is a different artifact from one that decides nothing quietly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
1ce7e9db9b |
fix(cpg): build php through the frontend and retry while it drops files
The intermittent empty-graph builds that cost four manual retries to measure
WP Statistics at all.
First, a correction: the explanation this repo carried was wrong. It said
php-parse's stderr banners interleave into block-buffered stdout and corrupt
the JSON. Capturing the exact bytes php2cpg receives -- by putting a logging
wrapper on `php` -- shows its input arriving complete and valid: two
well-formed JSON documents, 5,176,594 bytes, byte-identical across runs, with
no banner anywhere inside them. The defect is in php2cpg's own reading of that
input, and nothing outside it repairs a given attempt.
Three fixes were tried against the old theory and each changed nothing, which
is how the theory died:
* an unbuffered PHP shim via --php-parser-bin 0 of 6 builds usable
* compacting the JSON, 5,176,594 -> 943,028 bytes 0 of 6
* -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 0 of 6
What is true is that the failure is INTERMITTENT, and that `joern-parse` is
both worse at it and silent about it. Measured on one WordPress slice:
joern-parse 5 of 8 builds usable, and says nothing when they are not
php2cpg 11 of 12 builds usable, and names every file it dropped
So php builds through the frontend directly and retries while it reports
dropped files. Retrying is sound only because the failure is intermittent AND
reported -- retrying a silent failure would be superstition; retrying a
reported one is just using the report. It gives up after four attempts and
reports what is still missing rather than pretending a partial graph is whole.
wp-vuln through the driver before: 4+ manual retries to get one build
after: 6 of 6, 3 entailed findings each time
joern-parse remains the path for every other language, and the fallback when
no frontend is on PATH.
843 tests pass; gates byte-identical; mwwpform, pmpro and VAmPI unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
d6908f18ba |
feat(cpg): the hook half — CVE-2022-25148 found across an add_filter/apply_filters edge
The last of the three WordPress flaw classes 5.8 measured as missed. The
value crosses a registry inside WordPress core:
add_filter('wp_statistics_current_page', array($this, 'set_current_page'));
return apply_filters('wp_statistics_current_page', $current_page);
php2cpg drops the callback argument entirely -- the CPG holds
`add_filter("wp_statistics_current_page", )` -- so the edge cannot come from
the graph at any price. It comes from the source text instead, where both ends
are literals, via `mapping.php_hook_callbacks`.
vulnerable fixed
wpstatistics 3 entailed, incl. pages.php:225:record 2, that one absent
mwwpform 1 ($this->attachments) 0
pmpro 18, incl. CVE at :936 1
VAmPI 4 unchanged
The correctly-prepared sibling query one line below the CVE is not flagged on
either side.
Four things this cost, each measured rather than guessed:
* Half one must be asked per ARRAY KEY, not per callback. "Does an
unsanitized value reach this callback's return" is TRUE on both sides: the
fix escapes `type` and `id` and leaves `search_query` alone. A
callback-level answer flags the fixed side as readily as the vulnerable
one, which is a leak, not a detection. The key is a literal at both ends,
so it joins the way the hook name and the field name do -- the same idea
for the third time.
* The sanitizer test had to stop crediting a sibling's cleansing. The
vulnerable query is ONE concatenation holding both `esc_sql($page_uri)`
and the injectable `{$current_page['id']}`, so asking whether a
sanitizer's name appears in a path element's text marked the whole flow
sanitized. An element must now BE the cleansing -- a call to it, or its
name as a string literal, which keeps `array_map('esc_sql', ...)` working.
* Hook seeds are repository-wide where field seeds are file-scoped. A hook
registry is global by construction: the callback is in one file and its
`apply_filters` in another. The query returned nothing until those two
scopes were separated.
* `sourceKind` had to be REPORTED, not inferred. By the time a row reaches
Python a hook source reads as `$current_page["id"]` and a parameter as
`$args`, and `_usable_flows` discards parameter flows whenever a modelled
one is present -- so seven sanitized `$_SERVER["REQUEST_URI"]` flows
displaced the CVE between the query and the verdict. That preference then
had to move PER SINK SITE: applied across a region it decides one site's
question with another site's evidence, and PMPro's CVE at :936 vanished
from a region still reporting fifteen other findings.
`verdict` now delegates to `verdicts` rather than running its own selection
over the union, because two copies of "which flow decides" is exactly the
duplication this module has been bitten by before -- and had been again: the
per-site preference did not reach the single-verdict path, so the two entry
points disagreed about the same rows.
841 tests pass; gates byte-identical; VAmPI unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
dcdbe322a1 |
feat(cpg): taint a field where it is assigned — the field half of 5.11
`_delete_files()` takes no parameters. CVE-2023-6559 arrives as a FIELD,
written by one method and read by another:
public function __construct( ..., array $attachments = array() )
$this->attachments = $attachments; // half one ends here
protected function _delete_files()
foreach ( $this->attachments as $file ) // half two starts here
unlink( $file );
Both halves were already in the graph. Only the join was missing, and the key
it joins on is the literal `$this->attachments`, written identically at both
ends -- so this is a second question asked with the query that already exists,
not an edge a frontend has to invent.
A read of `$this->F` is a source when some assignment to the same literal
`$this->F` in the same file is fed by an unsanitized flow from a seed. Scoped
three ways so a field name common to every class ever written cannot become a
wormhole: only fields the sink scope reads, only assignments in the same file,
and the right-hand side must itself be reachable from a source.
The receiver could then leave `parameterNodes` -- the precision fix refused
two commits ago, now sound, because the path it was standing in for is carried
by the field it actually travels through:
before after
mwwpform vulnerable 1, from `this` 1, from $this->attachments
mwwpform fixed 0 0
pmpro vulnerable 15 -- 4 named, 11 receiver 18 -- all named or field
pmpro fixed 1, receiver-sourced 1
VAmPI 4 entailed 4 entailed, unchanged
CVE-2023-23488 is still at class.memberorder.php:936 and CVE-2023-6559 at
class.mail.php:259. Every finding now names the value it followed instead of
naming the object.
Three things cost a measurement each, recorded in the task:
* A summary must carry the SANITIZATION status of the half it summarises,
not merely its reachability. Without that clause,
`$this->sqlQuery = "..." . esc_sql($x) . "..."` reads as tainted -- and
PMPro builds most of its queries that way, so the fixed side went from 1
finding to 13. That is the pair no longer separating, which is the only
number that matters.
* Two-phase `reachableBy` then `reachableByFlows` is SLOWER, not faster:
183s -> 256s. Running the analysis twice costs more than skipping path
reconstruction saves. Reverted.
* Memoizing per field rather than per file took it to 97s, about 1.6x the
pre-field cost for that class, because only fields some region reads are
worth the flow query.
`inScope` is factored out of `sinkCalls` so the field sources are scoped to
exactly the same place as the sinks, rather than a second copy of those three
branches drifting from the first.
841 tests pass; gates byte-identical; VAmPI unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
e41d6dd7ea |
docs(contributor-scan): 5.11 -- a detection strategy, not a verdict of impossible
"Structurally impossible" is a verdict, and a verdict is not a plan. Replacing
both structural-miss tasks with the one mechanism that covers them, because
they are the same shape: two halves of a path joined by a key that is a
literal in the source text.
hook: $_REQUEST -> ... -> return of set_current_page
key: 'wp_statistics_current_page' literal at both ends
return of apply_filters(key) -> ... -> $wpdb->get_row
field: $_POST -> ... -> assignment to $this->attachments
key: attachments a field name at both ends
read of $this->attachments -> ... -> unlink
Neither half is missing from the graph. Only the join is. So this is not a
request for a frontend to draw an edge it cannot draw -- it is a second
question asked with the query that already exists:
stage 1 the existing taint question against a synthetic endpoint: does
untrusted data reach a callback's return, or an assignment to
$this->F?
table built by reading literals, never inferring: add_filter('H', c)
against apply_filters('H', ...); $this->F = against $this->F
stage 2 does the apply_filters return, or the field read, reach a sink?
join on the key, with the RUNG carrying the join's uncertainty -- one
registered callback and a literal key entails; several callbacks or
a computed hook name corroborates, and the witness says which
"Any hook may reach any handler" is refused absolutely: a complete graph over
the plugin would entail everything.
No new dependency, no CPG mutation, no IR -- both stages ride the batch that
already runs, which is the point of expressing it as a query-level join.
It also recovers the precision fix 5.8 had to refuse. With the field half
working, _delete_files() is reached through $this->attachments rather than
through $this being tainted wholesale, so the receiver can then leave
parameter sources and take PMPro's 11 receiver-sourced findings with it.
5.8's numbers are the before-side of that measurement, and both observables
are pairs already pinned.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
435c14f3ec |
docs(contributor-scan): 5.8 -- WordPress class by class, including what it cannot ask
Three plugins, each pinned at a version whose advisory was read out of the
code, each measured against its own fixed commit:
unauth injection, sink in the receiving function pmpro FOUND
arbitrary file operation mwwpform FOUND
unauth injection across a WordPress hook wpstatistics MISSED
missing capability check -- cannot be asked
unescaped output -- cannot be trusted
Both hits are entailed by the graph alone -- no judge, no spend -- and both
disappear on the fixed side, because both fixes are calls the fact tables
carry: `$wpdb->prepare` and `realpath()`. The engine is not recognising a bug;
it is failing to find a cleansing call on a path it resolved, then finding one.
Precision is stated rather than flattered. PMPro's vulnerable side reports 15
entailed, 4 from a named parameter and 11 from the receiver, and one of those
survives into 2.9.8 at a query whose every fragment is `esc_sql`-wrapped.
The miss is the useful half, and it is checkable rather than inferred. WP
Statistics' CVE has a modelled source, a modelled sink and `esc_sql` as its
fix, and reports nothing on either side, because the file holding the sink has
zero superglobals, makes no call into the class holding the source, and is
joined to it only by
add_filter('wp_statistics_current_page', array($this, 'set_current_page'));
return apply_filters('wp_statistics_current_page', $current_page);
Both ends name the callback with a string, so no frontend can draw the edge.
Most plugin data travels this way. What the engine reports there instead is two
other sites, the same class at the wrong lines, identical on both sides -- and
not noise: upstream annotates both `// phpcs:ignore WordPress.DB.PreparedSQL`
and later rewrote one with `$wpdb->prepare`.
Two classes could not be asked at all, which is not the same as a miss.
`dominance_specs(language="php")` is empty, so missing-capability is never
asked; and `output_encoding` shares the one flat sanitizer list, which holds
`esc_sql` and `intval` but not `esc_html` -- wrong in both directions, so no
XSS verdict from this engine currently means anything.
5.10's root cause is recorded too: php2cpg's parser bridge, a race rather than
a scale limit, and the mitigation is per-file exclusion. Adds 5.11 (the hook
edge, with CVE-2022-25148 as its observable) and 5.12 (taint a field where it
is assigned, with CVE-2023-6559 as its guard).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
e0442d146c |
fix(cpg): a graph the frontend emptied must not read as a clean repository
Scanning a two-file WordPress slice returned 64 regions examined, zero
findings and zero degradations. The graph it examined had no methods in it.
php2cpg 4.0.623 feeds several files to one `php-parse` process and reads its
stdout and stderr merged. php-parse writes a `====> File <next>:` banner to
stderr the moment it starts the next file, while the previous file's JSON --
megabytes of it -- is still draining from a block-buffered stdout. The banner
lands inside the JSON, ujson reports
expected json value or } got "=" at index 5011645
and the frontend drops that file. Reproducible on real plugin code, and the
size that matters is one file's AST dump, not the repository's:
class.memberorder.php alone (1834 lines) -> 142KB CPG, no failures
the same file + a two-line file -> 5.7KB CPG, BOTH dropped
One oversized file empties the whole graph. `joern-parse` then does not
propagate any of the frontend's warnings: 13 lines of output, exit status 0,
"Successfully wrote graph". This is what made a 637-file plugin unscannable
and reported it as clean -- it was never a scale limit.
Two fixes, because the failure arrives on two different paths:
* `_unparsed_files` reads the per-file warnings the frontend does emit, and
the scan reports them as `files_unparsed`. This covers the direct-frontend
path where the warnings are visible.
* The batched query now carries a `__census__` of the graph -- method and
file counts -- under a key no request id can collide with, and a scan over
a graph with no methods degrades as `cpg_empty`. This covers joern-parse,
which shows nothing. It rides the batch rather than costing its own
invocation, because avoiding a JVM start is the entire reason batching
exists.
Neither invents a verdict. There is no rung for "the graph was empty" because
the ladder labels verdicts and none was produced, which is exactly why it has
to be a degradation.
pmpro two-file slice before: 0 findings, degradations=[]
after: 0 findings, degradations=[cpg_empty]
Gates byte-identical; 840 tests pass.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
37a6594703 |
fix(mapping): a PHP method with a visibility modifier is a region
`^\s*function` matched a bare declaration and nothing else, so every method
written the way modern PHP writes them -- `public function`, `protected
function`, `private static function` -- was invisible to the region model.
The effect is not subtle on class-based plugins. MW WP Form's
`class.mail.php` produced ONE file-level region spanning 272 lines instead
of 16 method-scoped ones, and its `protected function _delete_files()` --
which holds CVE-2023-6559 -- was not a region at all. Paid Memberships Pro
happened to work only because that particular file omits the modifiers.
class.mail.php 1 file-level region -> 16 method-scoped regions
Also adds the MW WP Form checkout as the PATH family's plugin, pinned at
3bdd33d (5.0.3) with CVE-2023-6559 verified in the code:
protected function _delete_files() {
foreach ( $this->attachments as $file ) {
if ( file_exists( $file ) ) {
unlink( $file );
and the fix at 02ebc67 replacing the existence check with `realpath()`,
`is_file()` and a directory-confinement test. `unlink` is a modelled path
sink and `realpath` a modelled sanitizer, so this is a pair the engine
should separate rather than merely flag -- which is what makes it worth
more than a single hit.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
96e7476c6d |
feat: the first real CVE, found in production plugin code
CVE-2023-23488 is detected at
classes/class.memberorder.php:936:getMemberOrderByCode -- the exact line
verified in the source, in unmodified Paid Memberships Pro 2.9.7.
A two-file slice carrying both ends of the CVE, 2885 lines, loads and
queries in 122 seconds and reports 16 entailed injection findings.
**The other fifteen are not noise.** The file holds 26 `$wpdb` query calls
and **zero** of them use `->prepare`; all 26 interpolate or concatenate a
variable. The engine reports 16 of 26, so it under-reports rather than
over-reports, and every sampled finding is a genuine unprepared query --
`get_row("... WHERE id = '$id'")`, `get_var("... WHERE ID = '" .
$this->user_id . "'")`. CVE-2023-23488 is simply the instance that got an
identifier, because it is reachable unauthenticated; the rest are the same
defect with different reachability.
That answers the question that prompted this: whether the system can find
one real CVE. It can, in production code, at the exact line, in two
minutes, for a cent. What it could not do was survive its own plumbing at
637 files -- four separate quiet failures on this one checkout, all between
the scan and the arbiter, none of them in the analysis.
What this does NOT settle is recorded with it: the full checkout still does
not load and that boundary is unmeasured; whether php2cpg resolves
`new X()` and `$this->method()` is still unknown, since this slice reached
the sink through a same-function flow rather than across the object; and
one CVE on one plugin is not the class-by-class table over several plugins
that 5.8 asks for.
On cutting the input with a model, proposed when size looked like the
blocker: the evidence argues for scoping rather than triage. Two of 637
files found the CVE, task 2.10 already scopes by what a project declares it
ships, and diff-scoping is this spec's thesis. A model choosing what to cut
adds a fifth quiet-failure surface -- drop the file holding the bug and the
scan reports clean with nothing recording why, which is the shape of all
four failures found today. The reverse direction is where it fits: ranking
which regions get the budget, not choosing what to remove.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
21fdb554aa |
fix(cpg): a query that could not be asked is not an empty answer
contributor-scan 5.7, and it was the wrong question.
**The 117k-line pmpro CPG does not load.** Importing it throws in a
ForkJoinParallelCpgPassWithAccumulator during overlay application, so every
query against it returns nothing. It is not a call-graph question at all.
Isolated by running the ARBITER against two prebuilt CPGs rather than
writing a sixth ad-hoc Scala probe, four of which had already failed on my
own mistakes:
small wpdb CPG get_order_by_code -> 1 model_entailed
pmpro CPG getMemberOrderByCode -> 0
Both have their source and sink in ONE function, both were asked with
parameter sources and depth 3, same arbiter and same spec. The CPG is the
only difference.
That clears the facts, which is worth stating: `$wpdb` sinks work, the
parameter-source rule works, and a PHP parameter reaching `$wpdb->get_var`
through a concat is entailed. Whether php2cpg resolves `new X()` and
`$this->method()` is still unmeasured -- it cannot be measured on a CPG
that will not load -- and is recorded as 5.10 along with finding the size
boundary.
**And the defect underneath it, which is the fourth quiet failure found on
this one checkout.** `query_batch` returned `{}` for a failure and `{}` for
a genuine empty answer, so a caller could not tell "could not ask" from
"nothing found". Every query on this scan failed at load and it reported
500 regions examined, no findings, and no degradation: a vulnerable plugin,
clean.
A failed batch now returns None, unparseable output counts as a failure
rather than an empty answer, and the driver records a `query_failed`
degradation naming the kind and the request count.
This is the failure mode the rung ladder structurally cannot catch. Every
other honesty mechanism here labels a VERDICT -- suspicion,
regions_unjudged, regions_unasked, stated family limits -- and when no
verdict is produced there is nothing to label.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
8579c10e7c |
docs: the WordPress checkout scans, and does not find its CVE
Reporting the result rather than the intention. Paid Memberships Pro 2.9.7 with CVE-2023-23488 is pinned, verified in the code, and scans cleanly: 637 files, 117k lines, 1577 regions, build 47s, query 74s, 140s total, no degradations. **The CVE is not detected.** Both ends are inside the budget -- the REST handler at position 290 as a recognised entry point, the `$wpdb->get_var` sink at 472 -- and the taint query runs for 23 seconds. So the gap is the middle: the flow crosses `new MemberOrder( $code )` into a constructor and then `$this->getMemberOrderByCode( $code )`, both dynamic in PHP. Whether php2cpg resolves either into a callee edge is UNVERIFIED; two attempts to query the call graph failed on script errors and were abandoned rather than guessed at. Recorded as 5.7, with the answer required before anything is concluded about the fact tables. Three engine defects had to be fixed before the scan could produce even this result, and each had previously reported the plugin clean for a different reason: 1577 regions, 0 scanned joern-parse failed where php2cpg succeeded 500 scanned, taint 0.05s a 1.49MB batch over Linux's 128KB argv cap 500 scanned, taint 21.7s the entry point was not recognised as one The middle one is the one to remember. Every honesty mechanism in this codebase labels a verdict -- suspicion, regions_unjudged, regions_unasked, stated family limits -- and there no verdict was produced to label, so a vulnerable plugin was reported clean with every counter agreeing. What is established: the WordPress engine path works end to end. What is not: detection on a real plugin. One checkout that does not find its CVE is not yet evidence either way about the facts, and 5.7 has to answer whether the call graph can carry the flow at all. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
922ed84053 |
feat(repos): a WordPress plugin with a verified CVE, and the WordPress DB API
contributor-scan 5.1's detection half. Paid Memberships Pro 2.9.7 pinned at
1a900be3, carrying CVE-2023-23488 -- unauthenticated SQL injection through
the `code` parameter of the /pmpro/v1/order REST route.
**Verified in the code, not taken from the advisory.** At this commit
classes/class.memberorder.php:936 reads
$id = $wpdb->get_var("SELECT id FROM $wpdb->pmpro_membership_orders
WHERE code = '" . $code . "' LIMIT 1");
and tag 2.9.8 replaces it with `$wpdb->prepare( "... code = %s ...", $code )`.
637 PHP files and 117k lines, so it is also the first PHP checkout large
enough to say anything about an envelope.
**WordPress never calls mysqli; it goes through $wpdb**, so the facts could
not have found this. Added qualified: `$wpdb->get_var`, `->get_results`,
`->get_row`, `->get_col`, `->query`, `->replace`. Qualified because a bare
`get_var` or `query` matches any method of that name anywhere -- the
generic-name trap that made `query.get` match every `.get(` in a repository
this morning. `->insert`, `->update` and `->delete` are deliberately absent:
they take arrays and WordPress prepares them itself.
**And the thing that nearly went wrong.** The obvious next step when
writing PHP facts is to add WordPress's sanitizers -- sanitize_text_field,
esc_html, esc_attr. Doing so would have made this CVE invisible, because
its handler calls `sanitize_text_field( $params['code'] )` and then
concatenates the result into a query. sanitize_text_field is a real
sanitizer that does nothing whatsoever about SQL injection.
`TaintSpec.sanitizers` is one flat list per language whose comment states
the assumption outright -- "a sanitizer that breaks one flow breaks it
whatever the sink's family" -- and PHP falsifies it. So only SQL-specific
cleansers are listed, which leaves XSS under-sanitized instead, and the
real fix is recorded as task 5.6: a discharge counts for the obligation it
discharges. That is task 2.6's finding arriving in the taint family, where
the dominance arbiter already reads kind from the facts and the taint spec
has no equivalent.
The end-to-end scan of this checkout is not yet reported: the first run
returned 1577 regions with 0 scanned, and the CPG builds standalone at
4.3MB with no errors, so the two do not yet agree. Numbers when they do.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
60d2a1a92c |
feat(model): an endpoint its contract declares open carries no obligation
contributor-scan 2.8. VAmPI's last two false positives were register_user
and login_user, both declared open by its OpenAPI spec: the arbiter was
reporting a missing guard on endpoints whose contract says there is
nothing to miss.
**The fix keys on provenance, not on the value, and that distinction is
the whole task.** The mapper says "public" in two very different
situations. An OpenAPI operation with no `security` block is DECLARED
open, because the spec is a complete contract. A Flask handler with no
`@login_required` is "public" only because nothing was found -- which is
precisely the bug this family exists to report. Suppressing on the value
alone would have silenced the detector on its primary case, so
EntryPointRecord now carries `access_declared` and only a declaration
removes the obligation.
WordPress supplied the second source of declared access on the same day --
wp_ajax_nopriv_ for a logged-out handler, wp_ajax_ for a logged-in one,
permission_callback => __return_true for an open REST route. A rule resting
on one framework's convention would be a special case; two make it a shape.
vampi entailed 6 -> 4
access-control precision 2 of 4 -> 2 of 2
known vulnerabilities 3 of 3 still entailed
access_control offered to 6 of 14 handlers, exactly the six the spec
declares authenticated
**And the guard that decides whether it is a real improvement:** the
access-control pair corpus, which nobody tuned this against, is unmoved --
pair_correct 3 and leak 1 across all nine, pair_correct 2 and leak 0.0%
across the seven from projects unrelated to VAmPI, identical to task 2.6.
The precision was not bought by silencing the family. Pair excerpts carry
no route manifest and no hook, so nothing in them is declared open and the
rule correctly does not fire.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
3e1d13e4b6 |
feat(reports): the model's own suspicion band is a proposal too
contributor-scan 2.17, completing Req 5.5. Task 2.11 made static pattern matches proportionate and left the model layer's own suspicions alone, reasoning that a proposer had reasoned about them. That does not survive inspection. A model suspicion is what the enumerator proposed at a site the graph could NOT decide, affirmed by a judge with no coverage there -- and its own rationale says exactly that: "the model resolved no flow here; its sink table may not cover this API". It is a pattern match with an opinion attached, and it reads like a reasoned claim only because a model produced it. vampi 60 findings, 61 sections -> 26, report 1085 lines -> 446 libpng 198 findings, 39 sections -> 30 **What it costs, which is the number that decides whether the rule is right.** For each known vulnerability the recipe declares, take every finding row at its location and ask whether any survives the rule. All three of vampi's -- the SQL injection and both documented BOLAs -- are still found at model_entailed and all three survive. Nothing known is hidden. Three vulnerabilities in one repository is also a statement about how small the evidence base is, and the artifact says so. The obligations checker sits at `suspicion` too and is NOT grouped: its output comes from a structural analysis, an obligated operation with no discharging guard, rather than from a guess. An earlier attempt at this rule filtered on rung alone and would have collapsed it; six tests failed and they were right to. "Pattern matches" is now "Proposed, not corroborated", since it holds two shapes. Model rows group by FAMILY rather than by rule, because their rule is always `model`. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
f488e625c2 |
fix(specs): the php2cpg "blocker" was the shim, not the frontend
Task 5.5 recorded that php2cpg emits a CPG containing only one PHP file --
the last alphabetically -- and called it a blocker for the WordPress
target. Rerun with a native php-cli, as asked, and it is not true.
php 8.3.6, no shim, same joern 4.0.623, same two files:
files: plugin.php, vuln.php
methods: mp_fetch_report, mp_save_report, mp_render_box, mp_helper,
get_user, safe_user, show, run
Both files, all eight methods, and no PHP-Parser banner warning at all --
the banner that looked like the thread to pull was a symptom of the same
thing and does not occur natively.
The cause was the workaround. This machine has no PHP interpreter, so the
shim ran each `php` invocation in its own container; php2cpg drives
PHP-Parser once per file, and that per-invocation isolation kept only one
file's JSON.
**The lesson is worth more than the fix.** Two independent reproductions
agreed, and both were measured through the same broken instrument. A
consistent result is not a correct one when every measurement shares an
apparatus -- the same trap as a corpus that cannot see the defect it is
scoring, which is this session's other recurring finding.
And with the instrument fixed, 5.4 is demonstrably done. A two-file PHP
tree yields 10 function-scoped regions and 7 entailed findings in 15
seconds: SQL injection in both WordPress handlers, XSS in the shortcode
renderer, and all three of the original probe's. safe_user is correctly
absent, its mysqli_real_escape_string dropping it to corroborated. The
wp_ajax_nopriv_ handler ranks 1.0 and the plain helper 0.3.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
0ffa99b5e6 |
feat(model): one finding per sink site, and WordPress's registration model
contributor-scan 5.4. Both halves, and the second found a frontend blocker.
**One verdict per region was right for a pair and wrong for a file.** A
pair is one labelled function with one bug, so one answer is correct. A
region spanning a PHP file is not: the probe's SQL injection on line 5 and
command injection on line 13 produced ONE injection verdict, `system` won
it on flow length, and the SQL injection went unreported by construction
rather than by any failure of the analysis.
`taint.verdicts()` now returns one verdict per distinct sink site,
strongest first, and its first element is exactly what `verdict()`
returns -- so the pair path and every committed measurement resting on it
are unchanged, and the gates are byte-identical. Both entry points share
`_usable_flows` so neither can drift from the other.
php probe 2 of 3 -> 3 of 3
vampi 6 entailed, unchanged, now with lines on access-control
findings from task 2.7 (users.py:187, books.py:51 -- the two
documented BOLAs at their exact lines)
libpng 1 -> 2 entailed; the second was hidden by the collapse.
Neither is validated and both are in contrib/.
**WordPress registers handlers rather than decorating them, and the hook
name is the declaration.** `wp_ajax_nopriv_*` is WordPress saying
logged-out callers reach this, so it is public; `wp_ajax_*` fires only for
logged-in ones; `permission_callback => __return_true` declares a REST
route open. That is the declared-provenance signal task 2.8 needs,
arriving from a second framework. Plus every top-level PHP function as its
own region, without which a file is one region however many functions it
holds: 1 file-level region became 10 function-scoped ones.
**And a blocker, recorded as 5.5.** php2cpg emits a CPG containing only
ONE PHP file, the last alphabetically. Reproduced twice on independent
samples; both files parse cleanly under `php -l` and under PHP-Parser
directly, and joern-parse reports success either way. php2cpg logs "Found
warning in PHP-Parser JSON output" naming the survivor, and that warning is
PHP-Parser's own `====> File X:` banner appearing in what php2cpg expects
to be pure JSON.
Not isolated yet: this machine has no PHP interpreter, so php2cpg was
driven through a container shim. The banner comes from PHP-Parser rather
than the shim, which makes a shim-only cause unlikely but not excluded, and
rerunning with the native php-cli in the shipped image is the first step.
It blocks WordPress outright -- a plugin is many files, and a scan that
silently covers one of them and reports success is the quiet failure Req
5.6 was approved for.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
61811512a8 |
feat(facts): PHP semantic facts, validated against a real PHP CPG
contributor-scan 5.1, the first step toward the v0.1 WordPress target. Written against a PHP CPG rather than against PHP source, because the two do not look alike. Verified with joern-parse 4.0.623 / php2cpg on a probe: $_GET["name"] -> call `<operator>.indexAccess`, CODE $_GET["name"] mysqli_query($conn, $sql) -> call NAMED mysqli_query echo $_GET["msg"] -> call NAMED echo So sources match on code and sinks match on name, and every entry the probe did not exercise is marked unverified in the file -- `include` and `require` among them, which is PHP's most characteristic path bug and the first thing to check on a real checkout. Five families derive (injection, output_encoding, path, deserialization, untrusted_destination) with no sink that is its own sanitizer -- asserted by a test, because that bug class has now appeared three times in three matchers. `php` is added to the region model's languages, without which the facts would have reached nothing. The arbiter decides 4 of 4 on the probe: the SQL concat entails, its `mysqli_real_escape_string` twin drops to corroborated, `echo` XSS entails and `system` command injection entails. **A whole scan finds 2 of those 3, and the gap is the finding.** PHP files yield FILE-LEVEL regions because no PHP entry points are mapped, and the taint arbiter returns one verdict per region per family -- so a file-level region reports at most one injection finding however many it holds. `system` won on flow length and the SQL injection went unreported. That is the mirror image of task 2.9: that one is many regions collapsing to one site, this is many sites collapsing to one verdict, and it bites hardest exactly where PHP needs it not to, since a WordPress file is large and has no decorator for the mapper to find. Recorded as task 5.4 with WordPress's registration model (add_action, add_filter, admin-ajax, REST routes), which the mapper already handles in principle for other frameworks. A 13-line probe is not a WordPress result. What is established is that the seam works end to end for PHP; everything about scale and coverage is unmeasured, and there are still zero PHP pairs in a corpus of 263 python, 351 c_cpp, 47 javascript and 49 java. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
00e178d69e |
feat(reports): a report proportionate to what the engine can say
contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting that `memcpy` and `strcpy` are PRESENT in a library that uses them correctly throughout. That is the noise developers have learned to skip in every other tool, and shipping it costs the findings beside it their credibility. **Filtering on rung was the obvious rule and it was wrong.** The obligations checker emits at `suspicion` too, so hiding that rung would have hidden an entire detector -- six tests failed and they were right to. The discriminator is evidence_level: on libpng 163 of 198 rows are `static_corroboration`, on vampi 3 of 60 are, and the model layer and the obligations checker carry `suspicion` there instead. That is exactly the distinction the ladder exists to make -- an undecided CLAIM about a site is not an observation that an API exists. So a pattern-only rule is grouped once it has more than three sites, and keeps its full sections under that. One `python-flask-debug` is a finding a contributor can act on and must not be summarised away; fifty-four `c-unsafe-copy` are a property of the codebase. libpng 198 findings, 198 sections -> 39, 161 grouped into three lines vampi 60 findings, 61 sections -> 61, nothing grouped Counted, never dropped: findings.json stays complete because it is the record a baseline is compared against, and the grouped section says the model reached none of these and that its silence means no fact table covered the site rather than that the site is safe. What it does not fix, recorded rather than glossed: vampi still emits 61 sections for a 520-line application, 51 of them reasoned claims this rule does not touch. It fixes pattern noise, not volume. Also records what group 2 established about the three corpus asks, so none is started on a wrong assumption. Joern ships php2cpg but our fact tables have zero PHP families, so WordPress would parse and produce nothing -- fact tables first, corpus second. An older vulnerable libpng will not help for a reason no version can fix. Vibe-code repositories are the best fit and the cheapest route to a second true-positive checkout, because Python and JavaScript are the two languages whose fact tables are complete. And Group 8, for per-repository state a developer owns: a committed dismissal file the tool reads and never writes, dismissals as a split evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION -- the judge and residual prompts -- never to the fact tables. That boundary is the whole design: a prompt is how we ask, a fact table is what we claim, and the closed-loop leak measured on 2026-09-05 is why the distinction is not negotiable. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
9221ba8e40 |
feat(model): a bound on the data lowers the rung, it does not clear it
contributor-scan 2.12. c/injection and c/memory carried zero sanitizers, so no C flow could ever be discharged: every `argv -> strcpy` entailed however carefully it was bounded. Which sinks can be discharged this way is read from the sink fact's own CWE rather than declared: CWE-121 is a stack buffer overflow, so gets/sprintf/strcat/strcpy qualify. It is the only overflow CWE in the fact tables and it is C-only, so no Python or JavaScript sink is touched and the pair corpus is unaffected -- gates byte-identical. Not a sanitizer list, because `strlen` appears in the guarded and the unguarded case alike; the discharge is a guard, not a call. **The first implementation required the bound to DOMINATE the sink, and it found nothing.** libpng guards `strcpy(outname+len, ".png")` with `(len = strlen(inname)) > 250`, but that check sits on an else-if chain whose sibling branch skips it entirely, and the protection reaches the sink through an `error` flag: `++error` in the guarded branch, `if (!error)` around the copy. Joern's dominator set for that sink has 89 calls and correctly excludes the bound; `controlledBy` returns none either. Verified against the CPG before the rule was changed rather than after it failed. So the bound is searched in the method, and the consequence is corroborated rather than silence. That is the honest rung: a bound on this variable means the graph has NOT shown the copy is unguarded, and it has not shown it is guarded either -- off-by-one being how bounds fail. The judge is asked. Proving the flag-mediated case needs path-sensitive reasoning this arbiter does not do. libpng entailed 3 -> 1 The survivor is also a false positive: `strcpy(output + len, ".icc")` in iccfrompng.c, where `output = malloc(len + 5)` makes it fit exactly. That is a different discharge shape -- allocated to size, not bounds-checked -- and it is recorded as 2.13 rather than chased. Both shapes were found on one repository, and extending a rule until that repository is clean is the overfitting Req 8 exists to prevent; 2.13 requires a second C checkout with a genuine CWE-121 so the change can be measured for what it silences as well as what it clears. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
6dc25d77e4 |
feat(model): let the project say what it ships, and rank that first
contributor-scan 2.10. Every one of libpng's twenty-five entry-point regions was a `main()` -- twenty-three under contrib/, plus example.c and pngtest.c -- and those regions carried the fifty interprocedural taint requests that dominated a 300s query phase. A library has no entry points, so entry-point-driven analysis had nothing to anchor to except its own example programs. The task said to derive this rather than hardcode `contrib/`, and a project already declares it. Makefile.am names installed targets in `lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES` names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own word for "not installed" -- exactly the distinction wanted. Taking every `_SOURCES` would have called libpng's whole test suite shipped code, since contrib/libtests declares its sources the same way the library does. interprocedural regions 25 -> 1 taint query 300.1s -> 125.0s total query 311.9s -> 138.5s regions 121 -> 121, unchanged by design The surviving interprocedural region is contrib/tools/pngfix.c:main, which libpng genuinely installs. Two rules keep it honest. **Silence is not exclusion**: a project with no build file this can read gets None back and nothing changes, so vampi's numbers are untouched. **Not shipped is not unscanned**: those regions are still arbitrated function-locally, because a contributor editing a test must still get an answer -- what they lose is the expensive treatment and first claim on the budget. Entailed findings went 0 -> 3, and all three are false positives at contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe: outname is char[256] and line 325 guards with `if ((len = strlen(inname)) > 250)`. They were not introduced so much as uncovered -- a different callDepth meant a different sink set, and the 26 fgets false positives were masking them. The structural cause is recorded as 2.12: c/injection and c/memory carry ZERO sanitizers, so no C flow can ever be discharged and every `argv -> strcpy` entails however carefully it is bounded. The discharge here is not a call but a guard that dominates the sink, which is precisely what dominance.sc already does for access control and is wired to that one family. Also adds Group 7, five tasks for stopping the same false positive twice. Two of today's three false-positive classes were one bug in three matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in `fgets` -- each fixed locally and each next one surviving because nobody carried the fix across. A model may group counter-examples by cause and draft an edit; only a split-respecting measurement may adopt one, because the predecessor's improve lever admitted shapes taught by the holdout pairs they then recovered and turned a -0.059 Youden into +0.059. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
68b4e551f3 |
feat(model): the envelope checkout, and no to Joern server mode
contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform. 89,082 lines of C across 93 files, under a 2GB heap cap: CPG build 16.1s (25s cold) Joern queries 250.0s arbitration 122.0s (105 calls, $0.0122) whole scan 446s peak RSS 911MB regions 121, all arbitrated, 32 unasked CVE-2025-66293 out_of_scope, as the recipe declares Memory is not the constraint a fleet runner would hit: 911MB leaves room for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen while building this was concurrent JVMs on one desktop, not repository size. **Server mode: no.** The query phase splits taint 243.3s / config 10.7s over two invocations, and fixed overhead measured on the same CPG is ~13s per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL evaluation: a warm server removes 6% of the scan and none of the part that costs. The driver now records query seconds per kind, because a single total cannot tell startup from evaluation and the two answer this oppositely. The small repository pointed the other way. vampi's 34.3s over three kinds is roughly three startups and almost no evaluation, which made a warm JVM look worth having; the ratio inverts with size. That inversion is why the task was sequenced after a real repository rather than decided on a 520-line file. On whether one whole-repository CPG is the only shape for a horizontal fleet: no, and it is also not where the pressure is. The build parallelises and costs 16s; the query does not shard and costs 250s, so splitting a repository across runners divides the cheap phase -- and after 2.4 a shard boundary hides exactly the cross-module flows that matter. Scoping to a contributor's diff cuts the expensive phase instead. 911MB for 89k lines extrapolates to roughly 5GB at a million lines, which would exceed a standard runner, so the ceiling is real and further out than this checkout can locate. Two defects recorded rather than fixed, both invisible at vampi scale: 2.9, all 26 entailed findings are the same site reported 26 times, because regions that share a sink each emit their own; and 2.10, 22 of the 30 highest-ranked regions are libpng's `contrib/` example programs rather than the library. Also ignores Joern's `workspace/`, which a maintainer querying a CPG by hand from the repository root will create. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
d15985f695 |
fix(model): a guard discharges the obligation it is a guard of
contributor-scan 2.6. Access control entailed 7 of 9 VAmPI handlers,
including three that do authorize and three the OpenAPI spec declares
public. Five separate defects, each invisible to a corpus of one-function
excerpts and each found by asking the query what it actually matched.
**A trailing segment is not a match.** `query.get` was matched by its last
segment, so every `.get(` in the repository became a protected database
operation -- `request.headers.get('Authorization')` and
`request_data.get('username')` included. Four of the seven false positives
came from this, and so did the apparent detection of the known BOLA: it was
entailed because fetching the Authorization header had no dominating guard,
not because of the defect. A qualified token now requires its qualifier.
**Configuration is not authorization.** dominance_specs took every
obligation fact. `security_setting` operations are discharged by a VALUE,
which config_specs already claims them for; pooled in here they made
`app.run(...)` an operation requiring an authorization guard and put
thirteen tokens in BOTH lists -- an obligation that is its own discharge,
the same shape as the taint sink that used to sanitize itself.
**Input validation is not authorization.** `jsonschema.validate` was
counted as a guard, so handlers with no authorization check at all read as
guarded. No operation in the table requires validated_input.
**A guard discharges the obligation it is a guard of.** The largest one.
`orm-read` requires an identity_constraint or an ownership_check;
`token_validator` is a path_guard. Flattened into one list, authenticating
the caller discharged an object-level obligation -- a handler that
authenticates and then looks the record up by a PATH parameter read as
fully guarded. The facts state that relation and the spec was discarding
it; it is now carried and the engine asks per operation.
**A word boundary needs a word.** Found while fixing the above. Guard
matching wrapped every token in `(?<![A-Za-z0-9_])...(?![A-Za-z0-9_])`.
The identity token `['sub']` begins with `[`, and the character before it
in `resp['sub']` is `p` -- so the assertion could never hold. Every
identity token starting with punctuation was unmatchable, and the identity
constraint that separates a correct lookup from an IDOR was invisible.
Also: dominance rows were repository-wide, so the model's unguarded
`delete_user` was attributed to the API handler of the same name. Rows are
scoped to the region's file, which is what the rung's own justification
says -- the evidence is a file contradicting itself.
Measured against VAmPI's OpenAPI `security` blocks, which declare per
operation whether auth is required, and agree with the code:
vampi precision 28.6% (2/7) -> 50.0% (2/4)
known found 3 of 3, all model_entailed
And on the labelled access-control pairs, which nobody tuned this against:
all 9 pairs pair_correct 3 -> 3, leak 44.4% -> 11.1%
7 non-VAmPI pair_correct 2 -> 2, leak 42.9% -> 0.0%
Every detection lost was on a pair whose FIXED side was flagged too, which
scores nothing. No correct detection was traded for the precision.
get_by_title was flagged before it was in the recipe: its code carries the
comment "# Broken Object Level Authorization" and the README lists it, so a
real finding was being scored as a false positive. It is now the recipe's
third known vulnerability.
Both remaining false positives are endpoints the spec declares public.
Recorded as 2.8, with the reason the mapper's access_level cannot simply be
trusted: for decorator frameworks "public" means "no decorator found",
which is the bug itself.
Gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
0fcd5feade |
feat(model): a region for the code that is in no function
contributor-scan 2.5. VAmPI's `host='0.0.0.0'` is now reported exactly once, at app.py:17:<module>. Before task 2.2 it was reported eight times in eight files that do not contain it; after 2.2 it was reported nowhere. Two causes, one shape: nothing could ask about module-level code. **The main-guard marker is not a function.** `__main__` is what the mapper emits for `if __name__ == '__main__':`. Sent to the engine as a function name it matched nothing, because a CPG calls a file's top-level code `<module>` -- verified against the VAmPI CPG rather than assumed. The region model translates the marker, which is the one place that turns what the mapper meant into what the scanner asks. **A function-scoped file had no region for its module body.** Settings, route registration and globals live in no function, and every region's query filters by function name. A file-level region beside the handlers would report each handler's findings a second time, which is why the earlier rule dropped it as "a second full-file pass". A module-scope region PARTITIONS the file instead of overlapping it. That only became cheap after batching: before it, a region was a JVM. Two things this needed to be correct rather than merely present: - A module method spans its whole file, so the taint query's line-range containment made `<module>` the parent of every function in it. Under module scope, membership is direct -- otherwise the module region reports every handler's sinks again, which is the duplication the old rule was avoiding. - A module body is not a handler. The marker-derived region inherited the handler families and entailed a missing authorization check on the main guard: a finding with no possible remediation. Caught by the first repository run that had the region, and now covered by a test. bind address 8 wrong files -> 0 -> 1, at app.py:17:<module> regions 22 -> 25 entailed 8 -> 9 query time 37.4s -> 41.3s peak RSS 827MB -> 823MB cost $0.0067 -> $0.0068 Three more regions for four seconds and no more model calls: the budget is a total and was already spent, so regions_unasked rose 17 -> 20 -- those regions got the free pass and are reported as unasked, not as clean. `<module>` is Joern's Python name for a module body; JavaScript uses `:program`, so this is verified for Python only. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
ca7b5ba370 |
feat(model): a region that spans the flow, not just a function
contributor-scan 2.4, closing the no-go task 2.2 recorded. VAmPI's SQL injection is a request parameter reaching `get_by_username` in api_views/users.py, passed to `User.get_user` in models/user_model.py, interpolated into a query string and executed there. Source and sink are in different functions and different modules. Regions were per-function, so neither region saw both ends and the engine was silent on the most ordinary shape a real injection bug has. `reachableByFlows` was never the limitation -- it follows calls already. The limitation was the QUESTION. Sinks were filtered to the labeled function's own body, so a crossing flow had no sink to end at. Three changes, each with a reason it is sound rather than convenient: **An entry point is the trust boundary.** parameter_sources was off for repository scans because a repository's parameters are not all attacker input. An entry point's are, by definition -- that is what makes it an entry point. The driver turns it on only for regions the mapper found and named, and leaves it off for walker fallbacks and for named functions that are not entry points. **Sinks may sit below the handler.** taint.sc admits sinks in methods reachable from the labeled method through the call graph, bounded at ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is where these bugs live. Unbounded would make the question about most of the repository rather than about this entry point, and make it unaffordable. **The finding names the sink, not the asker.** A crossing flow ends in another file and another function, so Verdict carries the evidence's own `path:line:function` and the site uses it. Reporting `models/user_model.py:73:get_by_username` would name a place that does not exist: line 73 is inside get_user. Also: labeled methods are now resolved by name AND file. `get_user` exists in more than one module of a real repository, which a corpus of one-file excerpts could never surface. Measured on the same checkout and budget as 2.2: VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now entailed 7 -> 8 query time 40.9s -> 37.4s peak RSS 869MB -> 827MB cost $0.0067, unchanged Asking the bigger question cost nothing measurable here: file scoping and call-graph following pull in opposite directions and roughly cancel at this size. That is one 520-line repository, and VAmPI is now the repository this change was developed against -- the exact overfitting shape Req 8 exists to catch. The depth bound and the entry-point rule mean nothing general until a checkout nobody tuned against says so. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
177c7f3719 |
feat(model): first repository measurement, and the four things it found
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than as excerpts, with benchmarks/repos/measure.py recording build time, query time, arbitration time, peak memory, cost, unjudged regions, and whether each known vulnerability was found and at which rung. The corpus could not have found any of this. Every defect below needs a repository to become visible, and three of them were invisible behind a passing test suite. **The batch path was unreachable.** query_batch was implemented, tested and committed while CpgResult exposed only `run`, so the driver's capability check found nothing and fell back to one JVM per region-family: the first attempt at this measurement ran eighteen minutes on 520 lines, issuing `--param function=` one at a time. Every test called query_batch directly, so none of them noticed. CpgResult now carries run_batch and a test asserts the capability is reachable through the object the driver is handed. The earlier 275s->106s improvement attributed to batching came from region deduplication alone. **The budget stopped free work.** It bounds model calls, but was enforced by breaking the region loop -- spending a paid limit to stop unpaid CPG arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the graph entails for nothing, was never reached. It now withholds the judge only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent because unaffordable is not silent because decided), and the BOLA is entailed by the model layer. **A region with no function asked about the whole repository.** Sending function="" matched every hit in the tree and the driver attributed that answer to whoever asked: one `host='0.0.0.0'` in app.py became eight identical entailed findings in eight files that do not contain it. Queries now carry the region's file. Entailed findings 14 -> 7, query time 70.9s -> 40.9s. The single-request form stays unscoped so the committed pair measurements remain reproducible. **The JVM had no heap bound.** Joern claimed a quarter of physical RAM and runs were killed under memory pressure at different points, so the numbers described the host rather than the engine. -J-Xmx2048m by default, OPENULTRASAST_CPG_HEAP_MB to raise it. Also: arbitrated findings carried `function:line` where the report layer read `path:line`, so every entailed finding reached a contributor with a function name where its file belonged; and the witness line was parsed by splitting on a comma, which produced sites like `config.py:17): permissive literal '0.0.0.0':?`. Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end, 869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the only shape: VAmPI's SQL injection runs from a handler parameter into a function in another module, and a per-function region sees neither end. That is the ordinary shape of a real injection bug, so it is recorded as task 2.4 along with three smaller defects this run exposed. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
8c387306be |
feat(repos): pinned known-vulnerable checkouts, offline by default
The pair corpus scores excerpts: a vulnerable function and its fixed twin, handed to the engine with the answer already narrowed down. Nothing in it exercises a repository -- region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on fifteen lines still holds when the source, the sink and the guard live in three modules. benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char commit SHA, licence, and every known vulnerability with its file, function and line. `ousast repos` lists them; `--fetch` materialises one; CI does neither, and conftest clears the switch and redirects the cache so a developer with warm checkouts cannot read them by accident. Three rules the loader enforces rather than documents: - A branch or tag is rejected. Either would make the same recipe mean different code next month while the baseline it produced kept the old numbers. - `in_scope` must be stated per vulnerability, never defaulted. It decides whether "not found" is a miss or a non-question, which is the most important thing about the result. - A recipe that declares `detection` must name something in scope to detect. `ousast repos` does not trust a recipe either: against a materialised checkout it checks HEAD against the pin, that the licence file is there, and that each declared function appears above its declared line. That earned its place immediately -- the first libpng recipe named LICENSE.md, which is the filename at HEAD today and not at the pinned commit, where it is still LICENSE. Two checkouts, verified against the real repositories: - vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are already in the pair corpus, including the update_password miss diagnosed earlier, so a whole-repository run says directly whether a verdict survives the move off the excerpt. Its routes register through an OpenAPI operationId, so it exercises registration-based entry-point mapping rather than the decorator path. - libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at the parent of the commit that added the missing clamp. Its CVE is marked out of scope: an out-of-bounds read is not a family this engine arbitrates, and recording "not found" against it as a miss is the easiest way to make a measurement lie. Nothing is vendored; the recipe records the licence so an artifact can say what it ran against. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
8663c1a5b3 |
feat(contributor-scan): 1.1 regions from entry points, 1.2 the repository driver
Pair scoring could rely on a labelled function; a contributor cannot supply one -- which code is risky is the question they are asking. Regions now come from entry points, with a file-level fallback that says it is one, and each carries only the families its shape and language admit. Ranking uses what EntryPointRecord already decided (access_level, trust_boundary) rather than a second opinion computed from the same evidence. A family is offered only where its arbiter could answer: access_control needs a handler, so a library file with no entry point is not asked about obligations it has no siblings to be inconsistent with. The driver makes the two costs that were free at pair scale explicit. ONE CPG per repository, reused across regions -- a build is 5-30s on a forty-line excerpt and per-region builds would make this unusable. And a budget that is a TOTAL for the run rather than the enumerator's per-region cap of eight, spent highest-rank-first, with whatever it does not reach counted into regions_unjudged. A tool that silently analyses a tenth of a repository and reports a clean result is worse than one that says what it skipped. Two flaws the tests caught, both mine and both about ownership: the budget counted calls by reading a client's own usage log, so it silently never tripped for a client that keeps none -- the driver now counts its own spend through a wrapper; and the driver trusted its caller to have sorted, when the order the budget is spent in belongs to whoever holds the budget. The third was worse and only surfaced because a test failed for the right reason: the driver passed candidates=() and never enumerated, so a repository scan would have reported only what the arbiters establish unaided and never populated the suspicion band -- which supplied 15 of 26 findings in the pair measurement. It enumerates now. Sibling audit (Req 8.5) done before writing: family routing lives in exactly one production place, pipeline._arbitrate, dispatching on spec type. regions_for returns family names and the driver resolves specs, so that stays true. 750 passing, mypy clean over 96 files, gates byte-identical, zero orphans. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
fe45081b43 |
fix(ci): run ruff format, and record run 5 at 90% pair-correct
CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.
Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.
Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 90.0%
leak 12.3% 6.7%
detection -- 96.7%
cost ~$0.69 $0.0068
27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).
One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.
The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
c74e101d10 |
measure(model-grounded-detection): 86.7% pair-correct at half the hunter's leak
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 86.7%
leak 12.3% 6.7%
detection -- 93.3%
cost ~$0.69 $0.0067
Ten points of recall over the hunter, half the leak, a fifth of the model calls,
~100x cheaper. 26 of 30 pairs correct.
Four runs, and each step came from a measurement contradicting an assumption --
twice one I had stated:
1 73.3% the absence and config families had no pipeline path at all
2 73.3% routing fixed and the result IDENTICAL, which was the clue: the
arbiter was reached and its verdict thrown away, because every
corroboration was put to the judge as a taint question
3 80.0% residual question phrased for the arbiter that raised it
4 86.7% aiohttp Application(debug=True) and RedisStorage(httponly=False)
added to the settings table
The division of labour keeps moving the right way: model_entailed 9 -> 11 while
suspicion held at 15, so both findings gained in run 4 came from the MODEL, not
the LLM, and cost fell from $0.0126 to $0.0067 as work moved onto the free
deterministic path. That is the payoff curve for fact-table coverage, measured
rather than argued: closing a gap moves findings out of the LLM's column into
the model's, improving precision and cost at once.
Recorded corrections and limits, because the numbers are only worth what the
caveats are. The three silent config pairs were called corpus defects on a
similarity threshold borrowed from the OWASP minimal-fix analysis; it does not
apply to a slice that deliberately pairs a vulnerable function with a different
safe one, where 36% and 34% pairs both score pair_correct. Two were a settings
gap and now split. And permissive_values is a flat literal list with no keyword
association, holding both True and False -- debug=True is dangerous where
httponly=True is safe, and the model cannot tell them apart. It did not bite
here, and it is a real limit of the fact schema.
Four remaining misses, each with a named cause. Four runs cost about $0.04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
4e7fe65f17 |
measure(model-grounded-detection): the pipeline beats the hunter on both axes
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 80.0%
leak 12.3% 6.7%
cost ~$0.69 $0.0076
determinism none entailed band deterministic
Higher recall, half the leak, a fifth of the model calls, ~90x cheaper.
Three runs, and the middle one earned its keep by changing nothing. Run 2 fixed
family routing and returned numbers IDENTICAL to run 1 -- which was the clue
that the dominance arbiter was being reached and its verdict thrown away. Every
corroboration was put to the judge as a taint question ("a sanitizer appears on
the path -- does it suffice?"), meaningless for a missing guard, so the judge
said no and the finding was dropped. Run 3 phrases the residual question for
the arbiter that raised it and converts 2 of 3 access_control pairs from silent
to pair-correct, arriving at model_corroborated -- a rung that existed but had
never reached a finding.
This settles a question open since the start of the session, and reverses an
earlier answer of mine. "Detection has regressed" was measured with the
proposer disconnected: an arbiter compared against a detector, before four
wiring bugs were found.
Division of labour: 15 suspicion, 9 model_entailed, 2 model_corroborated. The
model arbitrates 11 of 26 findings, and every conversion from suspicion to a
model rung removes a leak and a model call at once -- the payoff curve for sink
coverage and interprocedural resolution, now measured rather than assumed.
Still silent: 3 config_secrets (the settings table is Flask/FastAPI-shaped
while the pairs are aiohttp) and 1 access_control still to diagnose.
Three full runs cost about $0.03 in total.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
6dbb0e6422 |
measure(model-grounded-detection): the wired pipeline halves the leak at 1/55th the cost
The first like-for-like comparison with the LLM hunter, on the same 30 vibe-py
pairs and the same metric.
hunter (K=5) pipeline (K=1)
pair-correct 76.7% 73.3%
leak 12.3% 6.7%
cost ~$0.69 $0.0126
determinism none entailed band deterministic
Half the leak, 55x cheaper, a fifth of the model calls, one pair short of the
hunter's recall (22 against 23 of 30). And understated: all six both_silent
pairs are access_control and config_secrets, which had no pipeline path to
their arbiters when this ran. A corrected run supersedes those rows.
The rung split says what is doing the work: 15 suspicion to 9 model_entailed on
the vulnerable side. The LLM supplies most of the recall; the model supplies
the precision, halving the leak by dropping claims it could contradict. That is
the division of labour the architecture was designed around and the first
evidence it pays.
It also revises an earlier conclusion of mine. "Detection has regressed" was
measured with the proposer disconnected -- an arbiter compared against a
detector. With the enumerator wired in, the comparison reverses on precision
and cost while holding recall.
And it makes the strategy concrete: the model arbitrates about a third of
findings today, and every suspicion converted to model_entailed is leak removed
for free and one fewer model call. Sink coverage and interprocedural resolution
now have a measured payoff rather than a hoped-for one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
cdedb372dd |
feat(model-grounded-detection): wire the detector -- the enumerator proposes, the model disposes
Everything before this built an ARBITER: something that judges a flow taint
reachability had already found unaided. Nothing proposed to it. That is why its
recall was never comparable to a detector's, and why comparing 41.3% entailment
against the LLM hunter's 76.7% pair-correct was comparing two different kinds
of thing. model/judge.py and enumerate_candidates were called by no production
code at all; every number so far came from a measurement harness.
The wiring turns on a distinction judge() alone conflated:
the model CONTRADICTS it has coverage at this site and says no -- a
sanitizer on the path, or a declared-safe sink shape.
The claim is dropped (Req 8.2).
the model CANNOT DECIDE no modelled sink or source reaches the site at all.
The model is SILENT, not negative, so the claim is
reported at `suspicion` (Req 8.4).
Treating silence as refutation is the tempting mistake and would be wrong here:
23 of the 39 misses in the entailment ceiling were sinks simply absent from the
fact tables, so "no flow" usually means the model cannot see it rather than
that nothing is there. The rung carries that uncertainty on the finding's face
instead of a confidence score nobody can act on.
An entailed site still costs zero model calls. The candidate budget is bounded
so a dense region cannot become an unbounded spend.
Smoke-tested end to end against a live endpoint: 2 pairs, detection 100%,
$0.0024. The full vibe-py measurement -- the first like-for-like comparison
with the hunter's 76.7% pair-correct and 12.3% leak -- is running.
727 passing, mypy/ruff clean, zero orphaned modules.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
1af3db37f6 |
feat(corpus): vendor 40 OWASP xss pairs as the owasp slice
output_encoding had exactly ONE pair in the whole corpus. It now has 41. 238 pairs were harvested from OWASP BenchmarkJava (2740 generated servlets, GPL-2.0, with a ground-truth CSV) by matching each really-vulnerable test to the safe test of the same category it most resembles -- adjacent tests come from one template, so they differ in the sink or the sanitiser rather than being unrelated programs. Median similarity 0.91, against the 64% of the older `sast` OWASP entries that the group-4 stratification flagged as different programs. Only 40 are vendored. One representative of each of the seven categories was run through the model BEFORE vendoring, and only xss distinguished: its fix adds ESAPI encoding, which the Java sanitizer facts recognise, so the vulnerable side entails and the fixed side drops to corroborated. cmdi, sqli, ldapi and xpathi put the sink in a helper class outside the servlet and resolved no flow at all; pathtraver found the flow but its "safe" twin carries no distinguishing call, so text similarity had matched a twin that is not semantically the fix -- a subtler form of the very defect the harvest was designed to avoid. Those 198 pairs are deliberately left out: adding pairs the model cannot read would inflate the corpus while making every number less interpretable. Evidence for the vendored slice is 3 of 3 distinguished, which is a small sample and is stated as such in the catalog header rather than implied to be more. The slice is marked SYNTHETIC in its header: it measures breadth across a weakness class and must never stand in for real-world performance. The corpus already shows a +16% overfitting gap, and a large generated slice blended into a headline would widen it invisibly. Two things recorded rather than hidden: BenchmarkJava's LICENSE is GPL-2.0 while the older sast entries declare GPL-3.0 for OWASP, which looks like an error and is not copied; and OWASP-Benchmark/BenchmarkJava already straddled the train/holdout split before these pairs (owasp-java-cmdi train, owasp-java-sqli holdout), so the new all-holdout pairs enlarge an existing straddle rather than create one. The two vendored-count guards did their job -- a corpus change could not land silently -- and are updated as the acknowledgement. Hygiene artifact regenerated: 277 -> 317 vendored pairs. 720 passing, mypy/ruff clean, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
fcb12e116a |
feat(model-grounded-detection): Java web sinks, and the post-closure-fix ceiling
Java shipped with injection, deserialization and crypto sinks only, so the path and output_encoding families had nothing routing to them by CWE and any Java pair in either was unreadable -- the gap hutool-cve-2018-17297 reported as "no path spec for java". Adds the JDK filesystem entry points, the servlet response writers, the outbound/redirect APIs, and the ESAPI / canonical-path sanitizers the OWASP suite itself uses. Documented APIs, provenance per entry. Post-closure-fix ceiling: 41.3% entailed (38/92) and 26.1% distinguished, up from 39.1% / 23.9%. path went 0/8 to 1/8 with its first distinguished pair; injection 29->31 entailed and 18->19 distinguished. Two points -- real, and the mechanism is understood, but the remaining gaps are data rather than machinery. Carrying a correction: the measurement harness routed config_secrets through TAINT rather than the constant-abstraction arm, so its 0/10 in this artifact measures the wrong arbiter and did so in both full runs. The harness is fixed; that family's real number is the earlier targeted re-measure, and this artifact's config row should not be read. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
0c71c956fd |
feat(model-grounded-detection): group 6, the execution tier as a seam not a build
The rung this project deliberately does not build. Clearwing already runs sandbox lifecycles, container pools, sanitizer images, PoC replay and stability classification, and the corrective brief concluded -- from this project's own defect history -- that reimplementing that machinery is where we go wrong. So model/execution.py decides eligibility, calls an adopted tool, and records what happened. A test asserts structurally that it imports no container or subprocess machinery and defines no build/run/compile/patch function, because the failure mode to prevent is this file quietly growing a sandbox. Eligibility is narrow in both directions: only a family a static model cannot arbitrate (memory safety in C, where the bug turns on runtime layout -- Req 10.3 forbids any web/logic family depending on this tier, and each now has taint, dominance or constant abstraction instead), and only a finding the model left undecided. The tier escalates uncertainty; it never re-checks a verdict. One deliberate asymmetry: a failure to reproduce is NOT a refutation. A sandbox that does not trigger a bug has not shown the bug is absent, only that this attempt did not trigger it, so the candidate stays where the model left it. Reading a failed reproduction as a clean bill of health is the same error class as reading an absent engine as "no vulnerability". With no adopted tool present -- the normal case -- the rung is never reached and clearwing_unavailable is recorded. All 16 tasks of model-grounded-detection are now done. 720 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
f1c341df6c |
feat(model-grounded-detection): group 5, corpus calibration and honest reporting
5.1 inverts what the corpus is for. Previously a pair the detector missed was a
DETECTOR failure, to be averaged away over K runs and chased with a prompt
change. The arbiter is deterministic: it will miss that pair the same way every
time, and the only thing that moves it is better modelling. So a pair the model
cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every
gap carries one even when the caller supplies none -- a bare gap teaches
nothing.
`covered` means the model reached a strictly higher rung on the vulnerable side
than on its fix. Flagging both sides equally is not coverage: it distinguishes
nothing, which is exactly what group 2 measured taint reachability doing on
every parameterised SQL pair. The harvest seam takes the fix commit and derives
the vulnerable side as its parent, with the fix diff as the oracle -- nothing
hand-labelled.
5.2 keeps three disciplines from the deleted publish layer, each because its
absence caused a real error here:
* per-slice rows always -- `render` RAISES without them. A mixed headline
once hid agent-written code scoring 44% against the development slice's 83%.
* the overfitting gap beside the headline, and ABSENT rather than zero when a
slice is missing: reading an unmeasured slice as "no overfitting" is the
flattering failure, not a neutral one.
* the rung on every finding, so a suspicion cannot be read as a result.
Run end to end against the committed ceiling artifact, the report says things
worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an
overfitting gap of +16%, and gaps grouped by family with their reasons --
injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no
verdict), untrusted_destination 5 (4 entailed on BOTH sides).
713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
A full re-measure with the closure-scoping fix is in flight; these numbers are
the pre-fix baseline and will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
91c0931a05 |
measure(model-grounded-detection): group 4 ceiling, 39.1% entailed against the flat IR's 2.2%
Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter that makes no model calls at all. Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10, path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the class no crash oracle can reach, arbitrated deterministically. A hypothesis tested and discarded. From the group-2 misses I diagnosed path's 0/8 as a missing req.url source token and said so. The probe -- the same 8 pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real cause is that the sink sits lexically inside an anonymous callback, so its enclosing method is the closure (<lambda>0) and the sinkMethod == function filter drops it; verdict(function="") on the same CPG returns model_entailed. And the obvious fix for that is wrong. Relaxing the filter raises the number with a FALSE witness: the flows found in the closure are `res -> readFileSync` and `this -> ...`, neither of which is the vulnerability, whose real path crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a miss into a wrong entailment is worse than the miss -- it is the arbiter asserting something untrue, the one failure mode this architecture exists to prevent. The correct fix is two-part and recorded: scope a sink to any method lexically nested in the labeled function, and scope parameter sources to that function's own parameters, then re-measure counting only real witnesses. Every miss is classified and none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source unmodelled with the idiom named. Two honesty items in the artifact. 16 of 92 pairs carry no function label and are arbitrated over the whole file, so a verdict can credit a different finding than the labeled one -- owasp-python-hash (CWE-327) is credited for a permissive set_cookie elsewhere in its file, on both sides. And config_secrets conflates three abstractions: permissive setting values (implemented), weak algorithm choice (facts already carry weak_literals, unwired) and hardcoded secrets (needs a credential-literal check). 694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
d9ceeb6a09 |
feat(model-grounded-detection): 4.2 constant abstraction, the third arbiter form
A configuration bug has no flow to follow and no guard to dominate.
CORS(app, origins="*") is dangerous because of the value it is SET TO, so the
arbiter evaluates the setting's arguments against the closed permissive set in
the obligation facts -- the third form beside taint reachability and guard
dominance, and the one Req 6.3 names.
ENTAILED a security setting is passed a literal in the permissive set
CORROBORATED the argument is computed, so constant abstraction cannot decide
it -- that residual is the judge's, never silently "safe"
(none) every literal is outside the set, or the call is not a setting
Two Joern node-shape bugs, both found by running it and both of the same class
as group 2's receiver bug -- the graph's indexing is not what the source text
suggests:
* a keyword argument (`origins="*"`) is an assignment-shaped node, so
isLiteral is false on the argument itself and the literal sits in its
subtree; literals are collected from the argument's AST now.
* Joern indexes the receiver at 0, positional arguments from 1, and NAMED
arguments at -1. Settings are configured overwhelmingly by keyword, so
argumentIndexGt(0) -- correct when counting arity for the taint shape test
-- dropped exactly the arguments that matter and every keyword-configured
setting read as "computed".
Live, deterministic, zero model calls:
create_app -> model_entailed CORS(app, origins = "*"): permissive literal "*"
safe_app -> no finding (origins = "https://app.example")
All three arbiter forms now work against a real CPG: taint reachability for the
flow families, guard dominance for absence, constant abstraction for config.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
fec67253f3 |
feat(model-grounded-detection): 4.1 guard dominance, 4.2 the missing sink facts
4.1 is the arbiter for the class no crash oracle reaches. An access-control bug
is the ABSENCE of a guard: no flow to follow, no crash to reproduce. What the
model can establish is a consistency violation -- an obligated operation no
discharging guard governs, in a file where its siblings are governed by one.
"Governed" turned out not to be one relation, and asking pure CFG dominance of
the operation gets two of three shapes wrong. Measured against the real engine:
identity constraint the guard is INSIDE the operation's arguments
filter_by(owner_id = current_user.id)
identity branch abort(403).controlledBy == [note.owner_id != current_user.id]
-- the check runs AFTER the fetch, so it never dominates
the operation, but it controls whether the value escapes
dominating guard a discharger-naming call that dominates the operation
That is why the query uses controlledBy rather than dominatedBy for shape 2.
Two bugs found by running it, both the same class as the sanitizer bug in group
2 -- part of a fact flattened, the part carrying the signal dropped:
* dominance_specs used only `calls` and `decorators`, omitting
identity_sources (current_user, request.user) and constraint_params
(owner_id, user_id) -- which is how an identity constraint is actually
written. The arbiter reported "no guard" on handlers that plainly had one.
request_sources and permissive_values stay excluded: they describe the
vulnerable shape, and including them would let the arbiter read a bug as
its own fix.
* matching operations by code containment yielded four or five desugared rows
per operation (the bare `tmp1.filter_by` receiver among them, carrying none
of the guards the real call has), so a guarded operation could be flagged.
Matching is by call name now.
Live, deterministic, zero model calls:
leaky -> model_entailed no discharging guard, while 2 sibling handlers are guarded
edit_note -> no finding (ownership check controls the denial)
list_notes -> no finding (identity constraint in the query)
4.2 authors the sink facts the go/no-go measured as its largest miss class (11
of 23): Django ORM raw SQL, and the four families that previously had no fact
routing to them by CWE at all -- path, output_encoding, untrusted_destination,
prototype. Each entry is a documented dangerous API of the named library with
its provenance recorded inline, and rule_ids left empty because a sink fact is
not a detection rule. Flagged for maintainer review as data, given the standing
"no LLM-authored patterns" constraint. The exact-set assertion written in task
1.1 to hold this gap open is updated rather than deleted, so it still fails if a
family ever loses its seed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
b8f8f727c6 |
feat(model-grounded-detection): group 3, the evidence ladder and the judge
3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The existing evidence_level carries `static_corroboration`, emitted by the flat IR overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as model_corroborated would claim a CPG arbitrated something it never saw, and rewriting every legacy finding to suspicion would be a product-visible regression nobody asked for. So `rung` defaults to suspicion and is raised only by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4: nothing rises above suspicion without a Verdict behind it. The witness travels with the rung, because a rung without its evidence is just a louder assertion. It reaches markdown, JSON and SARIF. 3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive outcomes: the model entails (report it, never call the model), the model contradicts (drop the claim, record why), the model cannot decide (report suspicion). The LLM gets one bounded typed question with tools=[] so it cannot search for the site, asked only about the residual the graph cannot settle -- whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow the CPG already found; it can never conjure one. Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while loop, or an import from the deleted round machinery. That test first matched the word "votes" in the module's own docstring, which is why it now inspects code rather than text. Verified live against a real CPG, not only scripted rows: vuln rung=model_entailed request.args["name"] -> db.execute(...) fixed rung=suspicion no source reaches this sink model calls made: 0 Both sides arbitrated by the graph with the LLM never consulted -- the thing the prior architecture could not do at any price. 678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
7387d2f190 |
measure(model-grounded-detection): task 2.3 reads GO on Joern
Injection slice, 50 pairs, ~30 minutes of real engine time.
model_entailed on the vulnerable side 54% (27/50) vs the flat IR's 2.2%
... on minimal-fix pairs only 61.5% (16/26)
pairs distinguished 32% overall, 38.5% on minimal-fix
determinism (Req 6.4) every verdict identical on re-run
cost median 32s/pair, max 58s
CPG build failures 1
The engine is not the constraint: all 23 misses trace to our own data model --
11 sinks absent from the fact tables (Django ORM objects.raw, self.query,
GraphQL resolvers), 11 where the sink is modelled but no source reaches it, and
one C translation unit that does not parse standalone, counted separately so it
can never read as "no vulnerability". That is exactly the result that justifies
adopting a CPG rather than building one.
The finding that changes the design: classifying what the 50 injection FIXES
actually do gives other_or_validation 37, bound_parameters 10, argv_list 2,
sanitizer_call 1. The dominant fix is a validation guard on the path, not a
sanitizer and not a sink shape -- and taint reachability can never separate
those pairs, because the flow still reaches the sink and what changed is that a
guard now dominates the path. The arbiter must be TAINT + DOMINANCE, which is
the same primitive task 4.1 builds for access_control, with the guard
vocabulary ported in task 1.1 as its lookup list. Two planned mechanisms
collapse into one, and 4.1 should now precede the remaining TaintSpecs.
Reported per stratum because only 26 of 50 pairs are genuine minimal-fix pairs:
12 are different programs (local-python-vulnerable is 7% similar with 83
changed lines) and 12 are large rewrites. A raw distinguished_rate would
measure corpus quality and report it as model quality.
Also corrected in the notes: the Flask-shaped source table is a real gap but a
minor lever -- adding Django patterns would gain a source for exactly one pair.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
2b973dc0db |
feat(model-grounded-detection): tasks 2.1-2.2, the Joern seam and the taint verdict
The whole engine integration is one subprocess boundary: joern-parse builds a cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between fence markers, because Joern's stdout carries a banner and a REPL prompt around whatever a script emits. Nothing is imported in-process, so the core install keeps dependencies = [] -- a test walks the seam's AST to assert it imports only the standard library and names no JVM binding. Fail closed everywhere. A build or query that errors, times out, or prints something unparseable yields no verdict and the caller reports suspicion. The taint verdict distinguishes this from absence deliberately: a query returning None means "the engine could not decide", which is not "there is no flow", and conflating them is how a missing engine silently becomes a clean bill of health. The verdict is where the ladder earns its keep. An unsanitized source->sink path is model_entailed -- the graph establishes it and the LLM's claim is no longer load-bearing. A path a sanitizer may break is model_corroborated: the flow is real, but whether the sanitizer suffices is the semantic residual the LLM answers, checked against this verdict rather than averaged over runs. The chosen flow is picked by a total order (shortest, then lexicographic), so two runs over one CPG cannot disagree -- Req 6.4 asserted directly. The query itself is the part our flat IR could not do: reachableByFlows walks the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is exactly the one-hop limit that held measured entailment at 2.2%. Capability detection reuses semantic/engines.joern_available rather than growing a second notion of "installed". The audit manifest is regenerated; the new modules are named by Req 4/5/6 rather than counted as orphans. Task 2.3, the go/no-go measurement, needs Joern installed and has not run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
9ed0e45c5f |
feat(model-grounded-detection): group 1, subtract the noise architecture
99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16 modules, ~4,100 lines) and six semantic/ modules removed, with the three gates byte-identical to 2026-09-06-gate-baseline.txt at every step. The sequence is what made it safe: vocabularies ported first (1.1), importers detached second (1.2-1.3), deletion third (1.4). Every import break during the deletion was therefore a detachment miss in the importer and was fixed there -- no module was restored to satisfy one. Both misses were the same shape and were fixed as a class: operations.py and shapes.py each imported trailing_name from the deleted variants.py, so it moved to semantic/ir.py, its natural home, and both importers changed in one edit. Ported before deleting: the closed guard vocabulary (verbatim, nothing else holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from the retained ruleset facts rather than copying them. Re-homed: families -> model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint, plus Region, labeled_family and labeled_spans. labeled_family no longer falls back to the LLM classifier -- an unlabeled pair is honestly unknown. Four deviations, each recorded in the spec notes with its reason. improve/ evolve.py and validator.py were stripped, not deleted: each mixes the mechanism lever (gone) with the deterministic rule-status loop `ousast improve` still runs, which has no LLM in it; a test now asserts the lever's absence by symbol so it cannot creep back. The learning CLI group became `model candidates`, because deleting it outright would have orphaned the retained enumerator by removing its only entry point. thinking moved from [learning] to [models]. A gap named rather than papered over: no retained sink fact routes by CWE to path, output_encoding, untrusted_destination or prototype -- the same hole the entailment ceiling diagnosed. The test asserts the seeded set and the gap set exactly, so task 4.2 must update it when it authors those sinks. The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness with no production caller and no CLI, alive only through its own test. The five named subsystems each carry an explicit decision. Manifest committed, with a test asserting the tree still matches it. Group 2 (the Joern go/no-go) has not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 4.8
|
9e811c1baf |
measure(model-grounded-detection): the entailment ceiling is 2.2%, and it is depth not a wall
The analogue of the candidate ceiling, one rung up: not "can a site be enumerated" (96.6%) but "can a deterministic abstract-interpretation model DECIDE the vulnerability without trusting the LLM". Offline, no model calls. model_entailed today 2.2% (2/92 web/logic pairs) model_corroborated, one-hop taint 24% (22/92) taint reachable, multi-hop 35% (26/74 flow families) candidate site enumerable 96.6% (for comparison) The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating, and it is the finding. The model cannot be a pure arbiter at v1. But the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed literal sink table (misses project wrappers and path/xss/prototype sinks), and a function-naming gap that eats 20/50 injection pairs. Deepening the taint to the candidate enumerator's depth-4 chain already moved injection 24% -> 48%. So the course is a layered arbiter, not a pure one: report model_entailed where the model proves it (deterministic, no LLM); let the LLM judge but check its claim against the resolved taint on the ~35% corroboration band (noise removed there); honest suspicion for the rest, which is where the deferred execution/differential tier earns its place. Modeling confirmed as abstract interpretation; SMT reserved for the memory/C tier; concolic avoided. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
66a51c674c |
docs(learning-harness): round 5, and what it says about the experiment itself
Round 5 re-ran round 1's content through the lever it should have used -- counterexamples, rendered to the detector as "Worked examples" rather than "Known non-issues, do not report these" -- and was rejected because one pair of eight moved, at p = 1.000. So the experiment I designed to give the demo lever a fair test could not give it one. Correcting my authoring error changed the reason for the verdict and not the verdict, because the rule cannot tell a lever that helps from a lever that does nothing. Five rounds, $2.17, zero decisions distinguishable from chance. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
78924316be |
docs(learning-harness): the reading of the first evolve rounds
Four rounds on injection, $1.76. Not one decision carried evidence, in either direction. Round 2 rejected a change because one pair of eight moved, at p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, both readings are what changing nothing looks like. I corrected an overstatement in the notes: after two rejections I wrote that the rule was near-incapable of accepting. Round 3 accepted. The accurate statement is symmetric and worse -- it accepts on noise as readily as it rejects on it. Escalated, not applied: Req 9.5's threshold rule should be replaced by an acceptance that survives the measured variance. The Pareto archive and the sign test both exist and neither decides anything. The honest caveat is that at a minibatch of 8 a reliability-gated rule would accept almost nothing either -- the corpus is the binding constraint, not the rule. And the finding that outranks the rounds, in the maintainer's words: the error is not the model, the error is the elsewhere. Eighteen defects this session, every one in this repository, three of them the same shape of fixing an instance and missing the class. The default assumption for an unexplained number is now that we broke it -- which puts pro's 46.7% against flash's 83.3% on our side of the ledger until shown otherwise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |