From 62052b952567ab39ed1794dfb2b6f0852939a7e4 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 10 Sep 2026 16:01:21 +0200 Subject: [PATCH] docs: where PHP detection stands, and the shortest path to a finishing scan Feeds the day's measurements back into the spec and reorders the roadmap around what is actually blocking. STATUS. Three CVEs found on real WordPress plugins -- CVE-2023-23488, CVE-2023-6559, CVE-2022-25148 -- each entailed by the graph with no model in the loop and each absent from its fixed side. Two of them require the two-stage join and would be invisible without it. The detector is not the problem. The repository is. A 637-file plugin builds in 76s to a 4.25MB graph, dominance and config answer, taint does not finish. Three blockers in the order they bite: cost (6.63 s/request), ranking (84% of regions share one rank, ordered alphabetically), coverage (access_control never asked for PHP, output_encoding untrustworthy while the sanitizer list is flat). And what is NOT a blocker, contrary to most of what this file said this morning: php2cpg. One genuine defect, 0.2% of files, detected and excluded and reported. Everything else attributed to the frontend was a local php shim proxying stdio, which cost four wrong diagnoses in a day. NEW TASK 5.14, and it is the cheapest thing on the board. 53% of taint requests are put to a family whose sinks are not in scope -- all 487 of PMPro's deserialization requests go to a repository that never calls unserialize. Skipping them cannot lose a finding, because a family with no sink cannot produce a flow. 4.6 hours -> 2.2, before any ranking work. ROADMAP reordered around one question: the shortest path to a scan of a real plugin that reports its CVE. Prune the impossible questions, then rank (an LLM job, and safe as one because ranking cannot manufacture a finding, with the closed-loop problem treated as a design requirement rather than a caveat), then attack the per-request cost only if those two are not enough. 857 tests pass. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/contributor-scan/tasks.md | 64 +++++++++++++++++++++++++++ .kiro/steering/roadmap.md | 54 +++++++++++++--------- 2 files changed, 96 insertions(+), 22 deletions(-) diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 91cc5d5..e7b9a38 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -842,6 +842,34 @@ - _Requirements: 4.2, 4.3, 9.1_ - _Depends: 5.10_ +- [ ] 5.14 Stop asking questions that cannot have an answer + - Half the taint budget is spent on questions whose answer is known in advance. Measured on the 500 regions + a PMPro scan actually examines: + + | family | asked | of those, the file holds no sink of that family | + |---|---|---| + | `deserialization` | 487 | **487 — every one** | + | `untrusted_destination` | 500 | 340 | + | `output_encoding` | 500 | 178 | + | `injection` | 500 | 165 | + | `path` | 500 | 140 | + | **total** | **2,487** | **1,310 (53%)** | + + At the measured 6.63 s/request that is **4.6 hours now, 2.2 hours pruned** -- and the cut costs nothing, + because a family with no sink in scope cannot produce a flow. This is not a heuristic. + - Two rules, and they differ in how exact they are: + - **Repository-level, exact at any `callDepth`.** If no file in the repository contains a sink of family + F, never ask F. That alone removes all 487 `deserialization` requests here -- a fifth of the budget -- + and PMPro simply never calls `unserialize`. + - **Scope-level, exact at `callDepth=0` only.** If the region's own file holds no sink of F, skip it. + Above zero the sink may live in a reachable method in another file, so this needs the reachable set + rather than the file, and it should be computed there or not claimed. + - Do this BEFORE the ranker. It is arithmetic rather than judgement, it cannot lose a finding, and it makes + every later measurement of ranking cheaper to run. + - Observable: taint requests issued per scan, and whether the query completes at all on `pmpro`. + - _Requirements: 4.2, 9.1_ + - _Depends: 5.13_ + - [ ] 5.12 The ranker knows one framework, and everything else lands in "unknown" - > "the ranker hardcodes wp nodes etc. which means it's not generalizeable. I'd like the LLM to be a > flexible layer here to avoid hardcoded / overly specific ranker implementations" @@ -930,6 +958,42 @@ - _Requirements: 4.4, 9.1_ - _Depends: 5.8_ +## Where PHP detection stands, 2026-09-10 + +**What works, measured on pinned checkouts with the CVE read out of the code:** three real CVEs found on real +WordPress plugins, each entailed by the graph alone with no model in the loop, each absent from its fixed +side. + +| | found at | sourced from | +|---|---|---| +| CVE-2023-23488 | `class.memberorder.php:936:getMemberOrderByCode` | `$id` | +| CVE-2023-6559 | `class.mail.php:259:_delete_files` | `$this->attachments` | +| CVE-2022-25148 | `class-wp-statistics-pages.php:225:record` | `$current_page["type"]` | + +Two of those needed the two-stage join (5.11) and would be invisible without it. The mechanism generalises: +one idea -- two halves of a path joined by a key that is a literal in the source -- covered a hook name, a +field name and an array key. + +**What does not work: the whole repository.** A 637-file plugin builds in 76s to a 4.25MB graph, `dominance` +and `config` answer, and `taint` does not finish. Three blockers, in the order they bite: + +1. **Cost.** 6.63 s per taint request (5.13). 500 regions is 4.6 hours; 53% of those requests cannot produce + a flow at all (5.14). +2. **Ranking.** 84% of regions share one rank, ordered alphabetically (5.12), so two thirds of the budget is + spent arbitrarily. +3. **Coverage.** `access_control` is never asked for PHP at all (5.9), and `output_encoding` cannot be + trusted while the sanitizer list is flat (5.6). + +**What is NOT a blocker, contrary to most of what this task file said this morning:** php2cpg. There is one +genuine frontend defect -- a `global` inside a closure produces a CPG Joern's own overlay rejects -- and it +affects 0.2% of files, is detected and excluded automatically, and is reported. Everything else attributed to +the frontend was a local `php` shim proxying stdio, which cost four wrong diagnoses in a day. + +**Honesty machinery, which is the part that held up.** Every failure above is reported rather than silent: +`cpg_empty`, `files_unparsed`, `cpg_sharded`, `regions_truncated`, `query_failed`, and a "What could not be +analysed" section that names them in the reader's terms. A scan that decides nothing and says so is a +different artifact from one that decides nothing quietly, and only the second is dangerous. + ## Group 6 — Regression baselines - [ ] 6.1 Repository baseline and delta diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index ace94a1..36861ff 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -51,32 +51,42 @@ So the ordering principle is: **make the output worth reading, then make it find behind an unreadable report changes nothing, and a second language before the report is fixed just doubles the noise. -## Now — make the output worth reading +## Now — make a repository scan finish at all -Everything here is in `contributor-scan`, group 2, and all of it is measurable on the two checkouts that exist. +The output work this section used to hold is done, and PHP went considerably further than it planned: three +CVEs on real WordPress plugins, entailed by the graph with no model in the loop, each dropping on its fixed +side. The detector is not the problem any more. **Finishing is.** -1. **2.17 — the model's own suspicion band.** vampi emits 61 sections for 520 lines; 38 are a candidate the - graph could not decide with a judge's opinion attached. Expected: 61 → ~23. Measure what it hides. -2. **2.9 — one defect is one finding.** 26 identical rows for one site. Dedupe on the site, keep the strongest - rung, record `reached_from`. -3. **2.16 — say what a family cannot decide.** Req 5.6. This is the anti-quiet-failure fix, and it matters - most for the teaching goal: an LLM handed a silent C scan will fluently explain why the code is fine. -4. **2.8 — a declared-public endpoint carries no obligation.** Removes both remaining vampi false positives. - Needs the access level's *provenance*, because for decorator frameworks "public" means "no decorator - found", which is the bug itself. +A 637-file plugin builds in 76 seconds and then the taint query does not complete. Everything below is +ordered by one question: what is the shortest path to a scan of a real plugin that reports its CVE? -## Next — reach the named targets +1. **5.14 — stop asking questions that cannot have an answer.** 53% of taint requests are put to a family + whose sinks do not appear in scope; every one of PMPro's 487 `deserialization` requests is asked of a + repository that never calls `unserialize`. Arithmetic, not judgement, and it cannot lose a finding. + 4.6 hours → 2.2. Do this first because it makes every later measurement cheaper to run. +2. **5.12 — rank, because 84% of regions currently share one rank ordered alphabetically.** Two thirds of the + budget is spent arbitrarily. Fifty well-chosen regions beat five hundred; combined with (1) that is the + difference between hours and minutes. The ranker is an LLM job, and it is safe as one because ranking + cannot manufacture a finding -- it decides what is looked at, never what is reported. + **The loop is closed and that is a design requirement, not a caveat:** a region the ranker excludes is + never arbitrated, so it never produces a label, so the ranker never learns it was wrong. An exploration + slice and a small fully-labelled corpus are what make the signal unbiased. +3. **5.13 — the 6.63 s/request itself**, if (1) and (2) are not enough. `callDepth=3` over 7,061 methods is + the term that makes a request cost seconds; the two-stage joins are NOT the cost (measured: 22%, and they + found two of the three CVEs). -5. **5.1 — PHP fact tables, then a WordPress checkout.** The shortest path to a target the maintainer named, - and the case this arbiter suits best. Sources, sinks and sanitizers for PHP; entry points for WordPress's - registration model (`add_action`, `add_filter`, admin-ajax, REST routes) which the mapper already handles - in principle. **Validate on pairs before pointing it at WordPress** — tables first, corpus second, because - the other order produces a confident silence. -6. **2.15 / 2.16 — settle `c/memory`.** Either a C checkout whose CVE runs through a sink we model, giving the - family its first true positive, or the family's scope narrowed and stated. Release criterion 4 needs one - or the other, and the second is cheap. libpng cannot answer the first; no version of it can. -7. **5.2 — a pinned vibe-code repository.** Still the cheapest *second* true-positive checkout, and the guard - against tuning everything on vampi. Below PHP now only because it is not a named release target. +Only after a scan finishes is there any point tuning what it finds. + +## Next — the families that cannot be asked + +4. **5.9 — PHP obligation facts.** `dominance_specs(language="php")` is empty, so missing-capability checks + are not asked on any WordPress scan. Not a miss; nothing was measured. +5. **5.6 — a sanitizer cleanses the family it cleanses.** The list is flat, so `esc_sql` reads as sufficient + for XSS and `esc_html` as insufficient for SQL. Wrong in both directions, so no `output_encoding` verdict + currently means anything. +6. **File the php2cpg defect upstream.** A `global` inside a closure yields a CPG Joern's own dataflow + overlay rejects, bisected to four lines. Worked around here (0.2% of files, detected and excluded and + reported), but it costs the WHOLE graph rather than the file, and other users will hit it. ## Then — close the deferred engine questions