marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: where PHP detection stands, and the shortest path to a finishing scan

Browse Source 
Feeds the day's measurements back into the spec and reorders the roadmap
around what is actually blocking.

STATUS. Three CVEs found on real WordPress plugins -- CVE-2023-23488,
CVE-2023-6559, CVE-2022-25148 -- each entailed by the graph with no model in
the loop and each absent from its fixed side. Two of them require the
two-stage join and would be invisible without it. The detector is not the
problem.

The repository is. A 637-file plugin builds in 76s to a 4.25MB graph,
dominance and config answer, taint does not finish. Three blockers in the
order they bite: cost (6.63 s/request), ranking (84% of regions share one rank,
ordered alphabetically), coverage (access_control never asked for PHP,
output_encoding untrustworthy while the sanitizer list is flat).

And what is NOT a blocker, contrary to most of what this file said this
morning: php2cpg. One genuine defect, 0.2% of files, detected and excluded and
reported. Everything else attributed to the frontend was a local php shim
proxying stdio, which cost four wrong diagnoses in a day.

NEW TASK 5.14, and it is the cheapest thing on the board. 53% of taint
requests are put to a family whose sinks are not in scope -- all 487 of PMPro's
deserialization requests go to a repository that never calls unserialize.
Skipping them cannot lose a finding, because a family with no sink cannot
produce a flow. 4.6 hours -> 2.2, before any ranking work.

ROADMAP reordered around one question: the shortest path to a scan of a real
plugin that reports its CVE. Prune the impossible questions, then rank (an LLM
job, and safe as one because ranking cannot manufacture a finding, with the
closed-loop problem treated as a design requirement rather than a caveat), then
attack the per-request cost only if those two are not enough.

857 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-10 16:01:21 +02:00
co-authored by Claude Opus 5
parent f496f41be1
commit 62052b9525
2 changed files with 96 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/contributor-scan/tasks.md
+64
View File
@@ -842,6 +842,34 @@
- _Requirements: 4.2, 4.3, 9.1_
- _Depends: 5.10_
- [ ] 5.14 Stop asking questions that cannot have an answer
- Half the taint budget is spent on questions whose answer is known in advance. Measured on the 500 regions
a PMPro scan actually examines:
| family | asked | of those, the file holds no sink of that family |
|---|---|---|
| `deserialization` | 487 | **487 — every one** |
| `untrusted_destination` | 500 | 340 |
| `output_encoding` | 500 | 178 |
| `injection` | 500 | 165 |
| `path` | 500 | 140 |
| **total** | **2,487** | **1,310 (53%)** |
At the measured 6.63 s/request that is **4.6 hours now, 2.2 hours pruned** -- and the cut costs nothing,
because a family with no sink in scope cannot produce a flow. This is not a heuristic.
- Two rules, and they differ in how exact they are:
- **Repository-level, exact at any `callDepth`.** If no file in the repository contains a sink of family
F, never ask F. That alone removes all 487 `deserialization` requests here -- a fifth of the budget --
and PMPro simply never calls `unserialize`.
- **Scope-level, exact at `callDepth=0` only.** If the region's own file holds no sink of F, skip it.
Above zero the sink may live in a reachable method in another file, so this needs the reachable set
rather than the file, and it should be computed there or not claimed.
- Do this BEFORE the ranker. It is arithmetic rather than judgement, it cannot lose a finding, and it makes
every later measurement of ranking cheaper to run.
- Observable: taint requests issued per scan, and whether the query completes at all on `pmpro`.
- _Requirements: 4.2, 9.1_
- _Depends: 5.13_
- [ ] 5.12 The ranker knows one framework, and everything else lands in "unknown"
- > "the ranker hardcodes wp nodes etc. which means it's not generalizeable. I'd like the LLM to be a
> flexible layer here to avoid hardcoded / overly specific ranker implementations"
@@ -930,6 +958,42 @@
- _Requirements: 4.4, 9.1_
- _Depends: 5.8_
## Where PHP detection stands, 2026-09-10
**What works, measured on pinned checkouts with the CVE read out of the code:** three real CVEs found on real
WordPress plugins, each entailed by the graph alone with no model in the loop, each absent from its fixed
side.
| | found at | sourced from |
|---|---|---|
| CVE-2023-23488 | `class.memberorder.php:936:getMemberOrderByCode` | `$id` |
| CVE-2023-6559 | `class.mail.php:259:_delete_files` | `$this->attachments` |
| CVE-2022-25148 | `class-wp-statistics-pages.php:225:record` | `$current_page["type"]` |
Two of those needed the two-stage join (5.11) and would be invisible without it. The mechanism generalises:
one idea -- two halves of a path joined by a key that is a literal in the source -- covered a hook name, a
field name and an array key.
**What does not work: the whole repository.** A 637-file plugin builds in 76s to a 4.25MB graph, `dominance`
and `config` answer, and `taint` does not finish. Three blockers, in the order they bite:
1. **Cost.** 6.63 s per taint request (5.13). 500 regions is 4.6 hours; 53% of those requests cannot produce
a flow at all (5.14).
2. **Ranking.** 84% of regions share one rank, ordered alphabetically (5.12), so two thirds of the budget is
spent arbitrarily.
3. **Coverage.** `access_control` is never asked for PHP at all (5.9), and `output_encoding` cannot be
trusted while the sanitizer list is flat (5.6).
**What is NOT a blocker, contrary to most of what this task file said this morning:** php2cpg. There is one
genuine frontend defect -- a `global` inside a closure produces a CPG Joern's own overlay rejects -- and it
affects 0.2% of files, is detected and excluded automatically, and is reported. Everything else attributed to
the frontend was a local `php` shim proxying stdio, which cost four wrong diagnoses in a day.
**Honesty machinery, which is the part that held up.** Every failure above is reported rather than silent:
`cpg_empty`, `files_unparsed`, `cpg_sharded`, `regions_truncated`, `query_failed`, and a "What could not be
analysed" section that names them in the reader's terms. A scan that decides nothing and says so is a
different artifact from one that decides nothing quietly, and only the second is dangerous.
## Group 6 — Regression baselines
- [ ] 6.1 Repository baseline and delta
.kiro/steering/roadmap.md
+32 -22
View File
@@ -51,32 +51,42 @@ So the ordering principle is: **make the output worth reading, then make it find
behind an unreadable report changes nothing, and a second language before the report is fixed just doubles the
noise.
## Now — make the output worth reading
## Now — make a repository scan finish at all
Everything here is in `contributor-scan`, group 2, and all of it is measurable on the two checkouts that exist.
The output work this section used to hold is done, and PHP went considerably further than it planned: three
CVEs on real WordPress plugins, entailed by the graph with no model in the loop, each dropping on its fixed
side. The detector is not the problem any more. **Finishing is.**
1. **2.17 — the model's own suspicion band.** vampi emits 61 sections for 520 lines; 38 are a candidate the
graph could not decide with a judge's opinion attached. Expected: 61 → ~23. Measure what it hides.
2. **2.9 — one defect is one finding.** 26 identical rows for one site. Dedupe on the site, keep the strongest
rung, record `reached_from`.
3. **2.16 — say what a family cannot decide.** Req 5.6. This is the anti-quiet-failure fix, and it matters
most for the teaching goal: an LLM handed a silent C scan will fluently explain why the code is fine.
4. **2.8 — a declared-public endpoint carries no obligation.** Removes both remaining vampi false positives.
Needs the access level's *provenance*, because for decorator frameworks "public" means "no decorator
found", which is the bug itself.
A 637-file plugin builds in 76 seconds and then the taint query does not complete. Everything below is
ordered by one question: what is the shortest path to a scan of a real plugin that reports its CVE?
## Next — reach the named targets
1. **5.14 — stop asking questions that cannot have an answer.** 53% of taint requests are put to a family
whose sinks do not appear in scope; every one of PMPro's 487 `deserialization` requests is asked of a
repository that never calls `unserialize`. Arithmetic, not judgement, and it cannot lose a finding.
4.6 hours → 2.2. Do this first because it makes every later measurement cheaper to run.
2. **5.12 — rank, because 84% of regions currently share one rank ordered alphabetically.** Two thirds of the
budget is spent arbitrarily. Fifty well-chosen regions beat five hundred; combined with (1) that is the
difference between hours and minutes. The ranker is an LLM job, and it is safe as one because ranking
cannot manufacture a finding -- it decides what is looked at, never what is reported.
**The loop is closed and that is a design requirement, not a caveat:** a region the ranker excludes is
never arbitrated, so it never produces a label, so the ranker never learns it was wrong. An exploration
slice and a small fully-labelled corpus are what make the signal unbiased.
3. **5.13 — the 6.63 s/request itself**, if (1) and (2) are not enough. `callDepth=3` over 7,061 methods is
the term that makes a request cost seconds; the two-stage joins are NOT the cost (measured: 22%, and they
found two of the three CVEs).
5. **5.1 — PHP fact tables, then a WordPress checkout.** The shortest path to a target the maintainer named,
and the case this arbiter suits best. Sources, sinks and sanitizers for PHP; entry points for WordPress's
registration model (`add_action`, `add_filter`, admin-ajax, REST routes) which the mapper already handles
in principle. **Validate on pairs before pointing it at WordPress** — tables first, corpus second, because
the other order produces a confident silence.
6. **2.15 / 2.16 — settle `c/memory`.** Either a C checkout whose CVE runs through a sink we model, giving the
family its first true positive, or the family's scope narrowed and stated. Release criterion 4 needs one
or the other, and the second is cheap. libpng cannot answer the first; no version of it can.
7. **5.2 — a pinned vibe-code repository.** Still the cheapest *second* true-positive checkout, and the guard
against tuning everything on vampi. Below PHP now only because it is not a named release target.
Only after a scan finishes is there any point tuning what it finds.
## Next — the families that cannot be asked
4. **5.9 — PHP obligation facts.** `dominance_specs(language="php")` is empty, so missing-capability checks
are not asked on any WordPress scan. Not a miss; nothing was measured.
5. **5.6 — a sanitizer cleanses the family it cleanses.** The list is flat, so `esc_sql` reads as sufficient
for XSS and `esc_html` as insufficient for SQL. Wrong in both directions, so no `output_encoding` verdict
currently means anything.
6. **File the php2cpg defect upstream.** A `global` inside a closure yields a CPG Joern's own dataflow
overlay rejects, bisected to four lines. Worked around here (0.2% of files, detected and excluded and
reported), but it costs the WHOLE graph rather than the file, and other users will hit it.
## Then — close the deferred engine questions