diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 08a3c63..4708f98 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -493,6 +493,36 @@ - _Requirements: 4.4, 8.1, 9.1_ - _Depends: 5.1_ +- [ ] 5.8 WordPress's popular flaw classes, on popular plugins + - > "in wp focus on some of their popular flaws and plugins. these are the keys to unlock practical + > software verification." + - One plugin is a checkout; a class is a claim. WordPress's recurring shapes are narrow and well + documented, and each maps to a family this engine already has: + - **Unauthenticated AJAX** — `wp_ajax_nopriv_*` reaching a `$wpdb` query. Declared-public plus injection, + both of which the mapper and the facts now carry. + - **Missing capability check** — a handler that acts without `current_user_can` / `check_admin_referer`. + This is the access_control family, and PHP has NO obligation facts yet, so it cannot be asked at all. + - **Unescaped output** — a stored option or meta value reaching `echo` without `esc_html`. Blocked on 5.6, + because the flat sanitizer list cannot say "escapes for HTML but not for SQL". + - **Arbitrary file operations** — a request value reaching `unlink`/`file_get_contents`. Facts exist. + - Pick plugins by installed base rather than by convenience, and pin each at a version with a verified + advisory the way `pmpro.toml` does -- the CVE read out of the code, not out of the advisory text. + - Observable: a class-by-class table over several plugins, saying for each whether it was found, missed, or + out of scope. That is the artifact worth teaching from; a single found CVE is not. + - _Requirements: 4.1, 4.4, 9.1_ + - _Depends: 5.7_ + +- [ ] 5.9 PHP obligation facts, so access control can be asked at all + - `dominance_specs(language="php")` returns nothing: PHP has no obligated operations and no dischargers, so + the access-control family is silent on every WordPress scan rather than wrong on it. + - The vocabulary is small and conventional -- `current_user_can`, `check_admin_referer`, `wp_verify_nonce`, + `is_user_logged_in` as dischargers; `$wpdb` writes, `update_option`, `wp_delete_post`, `wp_update_user` + as obligated operations -- and it is the single largest missing WordPress flaw class. + - Do it after 5.6, so a discharge counts for the obligation it discharges rather than for all of them. + - Observable: a missing-capability finding on a plugin that has one, and silence on one that checks. + - _Requirements: 8.1, 9.1_ + - _Depends: 5.6_ + ## Group 6 — Regression baselines - [ ] 6.1 Repository baseline and delta diff --git a/.kiro/specs/ir-consolidation/brief.md b/.kiro/specs/ir-consolidation/brief.md new file mode 100644 index 0000000..c2f652e --- /dev/null +++ b/.kiro/specs/ir-consolidation/brief.md @@ -0,0 +1,68 @@ +# Brief: ir-consolidation — stop maintaining an IR we did not need, and adopt one we do + +**Status:** brief only, unapproved. + +## Why now + +> "we should make sure our code base here doesn't explode. for example we have a custom IR. maybe vine +> (berkely) or another existing IR (rolf rolles had one) ... I know Ghidra has one we could steal" + +Two different questions live inside that, and they have opposite answers. Keeping them apart is most of the +value of this brief. + +## Question 1 — the source IR we already replaced and never retired + +`semantic/` is **3,218 lines, 16% of the codebase**: a hand-written IR (`ir.py`, `cst.py`) and a flat taint +engine (`taint.py`), plus the overlay that runs them. `model-grounded-detection` measured that IR against +Joern's CPG and the result was decisive — **2.2% entailment against 41.3%** — which is why the CPG was +adopted for the model layer. The flat IR was never removed, and still runs as a proposer in every scan. + +So the outgrowth is not something we are about to add. It is something already carried. + +**Vine and P-Code do not help here.** Both are *binary* IRs — Vine lifts executables (BitBlaze, UC Berkeley, +GPL-2.0, last significant activity 2009), and P-Code lifts machine code. Our source languages are Python, +PHP, JavaScript and Java, and the source IR we should adopt is the one we already have. The work is +subtraction, not substitution: + +- Can the CPG serve the overlay's job at equal or better recall? +- What does deleting `semantic/`'s IR and taint cost, measured on the pair corpus and both repositories? +- What has to stay? `obligations/` reads the same facts and is a different analysis. + +A candidate ~1,000-line deletion with a number attached, which is the only kind worth doing. + +## Question 2 — the abstraction we are missing, where an existing IR is exactly right + +`c/memory` has produced **zero true positives** and three distinct classes of false one. The reason is +structural and now stated in the report (Req 5.6): it models unbounded string CALLS, while real C overflows +are *arithmetic* — an index, a pointer offset, a `memcpy` whose size is computed. Every one of libpng's is, +and its library code never calls a modelled string API at all. + +"Can this expression exceed this allocation" is a value question, not a reachability one. **Rolf Rolles' +GhidraPAL carries a three-valued-logic abstract interpreter over Ghidra P-Code** — each bit definite 0, +definite 1, or unknown, with abstract transformers for `INT_ADD`, `INT_MULT`, `INT_AND`, the comparisons — +validated against concrete execution in its own tests. That is precisely the shape of analysis the C family +lacks, and it exists. + +It is also the honest answer to a decision this project already took. `model-grounded-detection` chose +abstract interpretation over SMT, and chose to build the model from code because execution cannot be +assumed. A bit-level abstract interpreter over P-Code is *still abstract interpretation* and still static — +what changes is the artifact: it needs a **built** library rather than its source. + +## What this brief must decide + +1. Whether the source-IR subtraction is safe, measured rather than assumed. +2. Whether the C arithmetic class is worth a second pipeline at all, or whether `c/memory` stays scoped to + string misuse and says so — which is where task 2.16 left it, deliberately. +3. If it is worth it: P-Code with a TVL domain, adapting GhidraPAL's transformers, versus writing one. Vine + is dormant since 2009 and is not the candidate; Ghidra is maintained and P-Code has tooling. +4. What a second pipeline costs in the one currency this project has been strict about — **not lines, but + the number of things that can silently return nothing.** This session found three separate quiet failures + in one PHP scan. A binary pipeline adds a build step, a lifter, and a domain, each of which can produce + an empty result that looks like a clean one. + +## The constraint that governs it + +Nothing here may grow the codebase without a measurement that justifies it, and the module audit +(`benchmarks/measurements/2026-09-08-module-audit.json`, currently 97 modules with no orphans) is the +instrument. A brief that ends in "adopt P-Code" and 3,000 new lines while `semantic/` still carries 3,218 +has made the problem worse, not better. **Question 1 should land before Question 2 starts.** diff --git a/.kiro/specs/ir-consolidation/spec.json b/.kiro/specs/ir-consolidation/spec.json new file mode 100644 index 0000000..b95efac --- /dev/null +++ b/.kiro/specs/ir-consolidation/spec.json @@ -0,0 +1,15 @@ +{ + "feature_name": "ir-consolidation", + "created_at": "2026-09-09T17:00:00Z", + "updated_at": "2026-09-09T17:00:00Z", + "language": "en", + "phase": "brief", + "approvals": { + "requirements": { "generated": false, "approved": false }, + "design": { "generated": false, "approved": false }, + "tasks": { "generated": false, "approved": false } + }, + "ready_for_implementation": false, + "depends_on": ["contributor-scan"], + "approval_note": "2026-09-09 brief drafted after the maintainer raised codebase growth and suggested adopting an existing IR (Vine, or Rolf Rolles' work, or Ghidra's). The brief separates two questions with opposite answers. The SOURCE IR is already carried rather than about to be added: semantic/ is 3218 lines, 16% of the codebase, holding a hand-written IR and flat taint engine measured at 2.2% entailment against the CPG's 41.3%, still running as a proposer -- and Vine and P-Code are binary IRs that do not help, so the work there is subtraction not substitution. The MISSING abstraction is the C arithmetic-overflow class, where GhidraPAL's three-valued-logic abstract interpreter over Ghidra P-Code is exactly the right shape and exists; Vine is dormant since 2009 and is not the candidate. Constraint recorded: question 1 lands before question 2 starts, or the brief makes the growth problem worse. Not approved." +} diff --git a/.kiro/steering/overview.md b/.kiro/steering/overview.md index 3c61877..12589fb 100644 --- a/.kiro/steering/overview.md +++ b/.kiro/steering/overview.md @@ -71,6 +71,44 @@ invocation per query kind. controlling control structure, so every guard question runs on CFG dominance instead. - **`execution_confirmed` has no production caller.** The rung exists and nothing reaches it. +## Why a corpus of benchmarks is not enough on its own + +This is the finding worth teaching, and it is not an opinion — it is what the gates say. + +**Twelve engine defects were found in one session by pointing the tool at three real repositories. The three +gates stayed byte-identical through every single fix.** A gate that does not move is a corpus that could not +see the bug. + +| defect | what the corpus saw | +|---|---| +| `resolve` matched inside `resolveUrl` (sanitizers) | nothing | +| `user` matched inside `users` (dischargers) | nothing | +| `gets` matched inside `fgets` (sinks) — 26 identical false positives | nothing | +| the model layer had zero production callers | nothing | +| `query_batch` implemented, tested, unreachable | nothing | +| a paid budget halting unpaid arbitration | nothing | +| a region with no function claiming the whole repository's answer | nothing | +| a guard discharging an obligation it is not a guard of | nothing | +| `joern-parse` failing where the frontend succeeds | nothing | +| a 1.49MB batch over Linux's 128KB argv cap | nothing | +| an array callable not recognised as a handler | nothing | +| one defect reported as twenty-six findings | nothing | + +The reason is structural rather than a gap in the corpus's coverage. **A pair is a labelled function with a +declared family and a known answer.** It cannot exercise: which region to look at, what to spend, whether the +capability is even reachable, whether two entry points share a sink, whether the question fits in an argv +slot, or whether a flow crosses a module boundary. Every one of those is a property of a *repository*, and +none of them is a property of a function. + +Three of the twelve are worse than invisible — they are **quiet failures**, where the tool produced a clean +report and every honesty counter agreed with it. The rung ladder labels a *verdict*; when no verdict is +produced there is nothing to label. That is the failure mode a benchmark score cannot express at all, because +the score is computed from the verdicts that exist. + +So the corpus is necessary and not sufficient: it is the **holdout** that stops per-repository tuning, and it +is the wrong instrument for finding out whether the tool works. Both instruments, or neither number means +anything. + ## The lesson that shaped the current specs Nine engine defects were found in one session by pointing the tool at two real repositories. **Three were the diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index bce2cfd..ace94a1 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -88,6 +88,24 @@ Everything here is in `contributor-scan`, group 2, and all of it is measurable o 10. **Group 3–4 — ship it.** Contributor output, the Docker image, the memory-family decision, retiring `evidence_level` now that the rung carries the meaning. +## Alongside — keep the codebase from growing into the problem + +> "we should make sure our code base here doesn't explode. for example we have a custom IR." + +`.kiro/specs/ir-consolidation/` (brief, unapproved) separates two questions with opposite answers: + +* **The source IR is already carried, not about to be added.** `semantic/` is 3,218 lines — 16% of the + codebase — holding a hand-written IR and flat taint engine measured at **2.2% entailment against the CPG's + 41.3%**, still running as a proposer. Vine and Ghidra P-Code are *binary* IRs and do not help here; the + work is subtraction. A candidate ~1,000-line deletion with a number attached. +* **The missing abstraction is C arithmetic**, and there an existing IR is exactly right. GhidraPAL's + three-valued-logic abstract interpreter over P-Code decides "can this index exceed this allocation", which + is the class `c/memory` cannot see and every libpng overflow is. Vine is dormant since 2009 and is not the + candidate. + +The ordering constraint is the point: **question 1 lands before question 2 starts**, or adopting an IR makes +the growth problem worse rather than better. + ## After — the feedback loop 11. **`finding-feedback-loop`** (design generated, awaiting approval). Its ledger and dismissals are useful