mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
Marius CiepluchandClaude Opus 5
e41d6dd7ea
"Structurally impossible" is a verdict, and a verdict is not a plan. Replacing
both structural-miss tasks with the one mechanism that covers them, because
they are the same shape: two halves of a path joined by a key that is a
literal in the source text.
hook: $_REQUEST -> ... -> return of set_current_page
key: 'wp_statistics_current_page' literal at both ends
return of apply_filters(key) -> ... -> $wpdb->get_row
field: $_POST -> ... -> assignment to $this->attachments
key: attachments a field name at both ends
read of $this->attachments -> ... -> unlink
Neither half is missing from the graph. Only the join is. So this is not a
request for a frontend to draw an edge it cannot draw -- it is a second
question asked with the query that already exists:
stage 1 the existing taint question against a synthetic endpoint: does
untrusted data reach a callback's return, or an assignment to
$this->F?
table built by reading literals, never inferring: add_filter('H', c)
against apply_filters('H', ...); $this->F = against $this->F
stage 2 does the apply_filters return, or the field read, reach a sink?
join on the key, with the RUNG carrying the join's uncertainty -- one
registered callback and a literal key entails; several callbacks or
a computed hook name corroborates, and the witness says which
"Any hook may reach any handler" is refused absolutely: a complete graph over
the plugin would entail everything.
No new dependency, no CPG mutation, no IR -- both stages ride the batch that
already runs, which is the point of expressing it as a query-level join.
It also recovers the precision fix 5.8 had to refuse. With the field half
working, _delete_files() is reached through $this->attachments rather than
through $this being tainted wholesale, so the receiver can then leave
parameter sources and take PMPro's 11 receiver-sourced findings with it.
5.8's numbers are the before-side of that measurement, and both observables
are pairs already pinned.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
170 lines
12 KiB
Markdown
170 lines
12 KiB
Markdown
# Pinned known-vulnerable checkouts
|
||
|
||
The pair corpus scores excerpts: a vulnerable function and its fixed twin, fifteen lines each, handed to the
|
||
engine with the answer already narrowed down. That measures arbitration. It cannot measure anything about a
|
||
repository — region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on
|
||
an excerpt still holds when the source, the sink and the guard live in three different modules.
|
||
|
||
These recipes name real checkouts so those questions can be measured instead of assumed.
|
||
|
||
## What a recipe is
|
||
|
||
One TOML file per repository, naming a url, a **full 40-character commit SHA**, the licence, and every known
|
||
vulnerability with the file, function and line it lives at.
|
||
|
||
```
|
||
ousast repos # what is pinned, and whether it is checked out
|
||
ousast repos --repo vampi --fetch # materialise one (network; CI never does this)
|
||
ousast repos --json # the same, for an artifact
|
||
```
|
||
|
||
Nothing here is vendored. The repositories stay in their own projects under their own licences; the recipe
|
||
records the licence so a measurement artifact can say what it ran against.
|
||
|
||
## Three rules, and why
|
||
|
||
**Offline by default.** A recipe is inert until someone asks for the network — `--fetch` for one run, or
|
||
`OPENULTRASAST_REPOS_NETWORK=1`. CI sets neither, and the test suite clears both. This is the same rule the
|
||
pointer pairs follow, for the same reason: a test suite that quietly reaches the network is a test suite that
|
||
fails for reasons that have nothing to do with the code.
|
||
|
||
**Pinned, not tracked.** `commit` must be a full SHA; a branch or a tag is rejected at load. A tag gets
|
||
re-cut and a branch moves every week, and either would make the same recipe mean different code next month
|
||
while the baseline it produced still carried the old numbers.
|
||
|
||
**A recipe declares what it can measure.** `measures` is `detection`, `envelope`, or both, and every known
|
||
vulnerability states `in_scope` explicitly — whether it belongs to a family this engine arbitrates at all.
|
||
There is no default, because the default would silently decide the most important question about the result:
|
||
|
||
> An out-of-bounds read in a C library is a fine envelope test and an impossible detection test. Recording
|
||
> "not found" against it as a miss is the easiest way to make a measurement lie.
|
||
|
||
## Verification
|
||
|
||
`ousast repos` does not trust a recipe. Against a materialised checkout it checks that HEAD is the pinned
|
||
commit, that the licence file is really there, and that each declared function appears above its declared
|
||
line in its declared file. A recipe that contradicts the code exits non-zero; a recipe that simply has not
|
||
been fetched does not, or CI could never run this command.
|
||
|
||
That check earned its place immediately: the first libpng recipe named `LICENSE.md`, which is the filename at
|
||
HEAD today and not at the pinned commit, where it is still `LICENSE`.
|
||
|
||
## What is pinned
|
||
|
||
| repo | language | size | measures | why |
|
||
|---|---|---|---|---|
|
||
| `vampi` | python | 520 lines, 11 files | detection | Small enough that a detection result is not confounded by budget or ranking. Its routes are registered through an OpenAPI `operationId` rather than decorators, so it exercises registration-based entry-point mapping. Excerpts of it are already in the pair corpus, so a whole-repository run says directly whether a verdict survives the move off the excerpt. |
|
||
| `libpng` | c | 91k lines, 101 files | envelope | Whether one CPG over a real C library is affordable at all: build time, query time, peak memory, and whether the ranker yields a sane number of regions. Its CVE is out of scope by construction. |
|
||
| `pmpro` | php | 117k lines, 637 files | detection, envelope | Paid Memberships Pro 2.9.7. CVE-2023-23488 is an unauthenticated SQL injection whose value reaches the sink inside the function that receives it -- the shape the PHP fact tables were built for. 2.9.8 wraps it in `$wpdb->prepare`, so the pair separates. |
|
||
| `mwwpform` | php | MW WP Form 5.0.3 | detection | CVE-2023-6559, an arbitrary file deletion. The fix adds `realpath()`, `is_file()` and a directory-confinement test, all of which the facts already carry, so this is a pair the engine can separate rather than merely flag. |
|
||
| `wpstatistics` | php | WP Statistics 13.1.x | detection | CVE-2022-25148, an unauthenticated SQL injection that crosses `add_filter`/`apply_filters`. Pinned because it is NOT expected to be found: a table of hits teaches nothing about where the engine stops. |
|
||
|
||
## WordPress, class by class
|
||
|
||
WordPress's recurring flaw classes are narrow and well documented, which makes them a fair test: each one
|
||
maps to a family this engine already has, so "not found" has to name a mechanism rather than shrug. Three
|
||
plugins, each pinned at a version whose advisory was **read out of the code** and not out of the advisory
|
||
text, and each measured against its own fixed commit.
|
||
|
||
| class | plugin, CVE | vulnerable side | fixed side | verdict |
|
||
|---|---|---|---|---|
|
||
| Unauthenticated injection, sink in the function that receives the value | `pmpro`, CVE-2023-23488 | entailed at `class.memberorder.php:936:getMemberOrderByCode`, sourced from `$code` | absent | **found**, and the pair separates |
|
||
| Arbitrary file operation | `mwwpform`, CVE-2023-6559 | entailed at `class.mail.php:259:_delete_files` | absent | **found**, and the pair separates |
|
||
| Unauthenticated injection, value crosses a WordPress hook | `wpstatistics`, CVE-2022-25148 | not reported | not reported | **missed**, structurally |
|
||
| Missing capability check | — | — | — | **cannot be asked**: PHP has no obligation facts |
|
||
| Unescaped output | — | — | — | **cannot be trusted**: the sanitizer list is flat |
|
||
|
||
### The two it finds
|
||
|
||
Both are entailed by the graph alone — no judge, no model call, no spend — and both disappear on the fixed
|
||
side because the fix is a call the fact tables already carry: `$wpdb->prepare` for one, `realpath()` for the
|
||
other. That is the distinction worth teaching. The engine is not recognising a bug; it is failing to find a
|
||
cleansing call on a path it resolved, and then finding one.
|
||
|
||
The precision is not clean, and saying so is the point. On PMPro's order class the vulnerable side reports
|
||
**15 entailed findings**, of which **4** are sourced from a named parameter and **11** from the receiver,
|
||
`$this`. One receiver-sourced finding survives into 2.9.8 at a query whose every fragment is wrapped in
|
||
`esc_sql` — a false positive, and a legible one: the flow reaches the sink through `$this->sqlQuery`, so the
|
||
sanitizer is on the fragments and not on the path.
|
||
|
||
Excluding the receiver was tried and reverted. It cuts that file to 6 findings, all named-parameter sourced,
|
||
and makes the PMPro pair separate cleanly — and it **loses CVE-2023-6559**, because `_delete_files()` takes
|
||
no parameters at all and its attacker-controlled path arrives as `$this->attachments`, assigned by another
|
||
method. Trading a verified CVE for findings merely suspected of being false is the wrong trade. The real
|
||
defect is field-sensitivity: a field should be tainted where it is assigned.
|
||
|
||
### The one it misses, and why
|
||
|
||
WP Statistics' CVE is an unauthenticated REST hit whose `current_page_id` is interpolated into
|
||
`$wpdb->get_row`. Every piece is in scope — a modelled source, a modelled sink, and `esc_sql` as the fix —
|
||
and the engine still reports nothing, on both sides. The reason is checkable rather than inferred:
|
||
|
||
* the file holding the sink contains **zero** superglobals;
|
||
* it makes **no** call into the class holding the source;
|
||
* the only connection between them is
|
||
|
||
```php
|
||
add_filter('wp_statistics_current_page', array($this, 'set_current_page')); // hits.php:43
|
||
return apply_filters('wp_statistics_current_page', $current_page); // pages.php:105
|
||
```
|
||
|
||
Both ends name the callback with a **string**. No frontend can draw that edge, so no dataflow engine can
|
||
cross it, and the honest description is not "we missed it" but "the graph does not contain the path". Most
|
||
plugin data travels this way. It is the single largest structural gap for PHP, and it is not a fact-table
|
||
problem.
|
||
|
||
It is also not a permanent one, and "structurally impossible" would be the wrong thing to leave here. Note
|
||
what is and is not missing: `$_REQUEST` reaching `set_current_page`'s return is a path the graph *has*, and
|
||
`apply_filters`' return reaching `$wpdb->get_row` is a path the graph *has*. Only the join between them is
|
||
absent — and the key it joins on, `'wp_statistics_current_page'`, is a string literal written at both ends.
|
||
The same is true of `$this->attachments` in MW WP Form, where the key is a field name.
|
||
|
||
So the strategy is two-stage taint with a link table: ask the existing query for each half against a
|
||
synthetic endpoint, build the table by reading literals rather than inferring anything, and join. The rung
|
||
carries the join's uncertainty — one registered callback and a literal key entails, several callbacks or a
|
||
computed hook name corroborates. Task 5.11 states it, with both pinned pairs as its observable.
|
||
|
||
What the engine reports on that plugin instead is two other sites — `getTop:406` and `TotalCount:443` — the
|
||
same weakness class at the wrong lines, identical on both sides, so the pair does not separate. They are not
|
||
noise: upstream now annotates both with `// phpcs:ignore WordPress.DB.PreparedSQL...`, and later rewrote
|
||
one of them with `$wpdb->prepare`. Right class, wrong sites, no discrimination.
|
||
|
||
### The two it cannot ask
|
||
|
||
`dominance_specs(language="php")` returns an **empty** mapping. There are no PHP obligation facts, so the
|
||
missing-capability class — `current_user_can`, `check_admin_referer` — is never asked, on any plugin, and
|
||
would report nothing however many were added. That is not a miss; nothing was measured. Recorded as 5.9.
|
||
|
||
`output_encoding` for PHP has sinks (`echo`, `print`, `printf`) and shares the one **flat** sanitizer list
|
||
with every other family. That list contains `esc_sql`, `intval`, `realpath` and `escapeshellarg`, none of
|
||
which escape HTML, and excludes `esc_html` and `esc_attr`, which are the only ones that do. So it is wrong in
|
||
both directions at once: it would accept `echo esc_sql($x)` as safe and report
|
||
`echo esc_attr(sanitize_text_field($_REQUEST['login']))` — the pattern WordPress documents as correct, and
|
||
which appears verbatim at `pmpro/includes/login.php:684` — as a finding. Until a sanitizer can say which
|
||
weakness class it cleanses, no XSS verdict from this engine means anything. Recorded as 5.6.
|
||
|
||
### What nearly invalidated all of it
|
||
|
||
The first PMPro measurement in this series reported 64 regions examined, zero findings and zero degradations.
|
||
The graph it examined had no methods in it.
|
||
|
||
`php2cpg` 4.0.623 reads its PHP parser's stdout and stderr as one merged stream, and the parser writes a
|
||
progress banner to stderr the moment it starts the next file — while the previous file's multi-megabyte JSON
|
||
is still draining from a block-buffered stdout. The banner lands inside the JSON, the file is dropped, and
|
||
`joern-parse` propagates none of it: thirteen lines of output, exit status 0, "Successfully wrote graph".
|
||
One oversized file empties the entire graph, so this was never the scale limit it looked like.
|
||
|
||
It is also a **race**, not a threshold. Five builds of the same two-file input: three produced a graph, two
|
||
came back empty. Before the batched query started carrying a census of the graph, two of every five scans of
|
||
that plugin would have reported a clean bill of health.
|
||
|
||
That is the argument for why a corpus of benchmarks is not enough on its own, made concretely. Every gate
|
||
stayed byte-identical through all of it. A corpus tells you a verdict is right; it cannot tell you the
|
||
verdict was reached by looking at anything.
|
||
|
||
## Baselines
|
||
|
||
Task 6.1 turns a scan of a pinned checkout into a committed baseline, so a change to the fact tables or the
|
||
queries can be read as findings gained, findings lost and rung movements (Req 10.1–10.4). Until then these
|
||
recipes exist to be measured against by hand, and the first such measurement is task 2.2.
|