From 9910dd18b9d7d9c68d5dd37fa4622785418bacd4 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 22 Sep 2026 09:53:52 +0200 Subject: [PATCH] measure(php): the query cost is the matched sink count, and the blocker is identified Open since September and re-measured nine times. The cost of a taint batch is the number of CALL NODES a family sink names match in the repository. Not the request count, not the declared sink count, not any request parameter. reachableByFlows is asked per matched sink, and PHP soutput-encoding sinks are echo and print, which a WordPress plugin writes thousands of times. Three controlled sweeps on one graph, built once. Batch size exonerated: 400 requests answer in 170 s, and cost per request FALLS with size because roughly 70 s is fixed JVM start and graph load. Request parameters exonerated: at 200 requests the full production shape, call depth 3 with parameter sources, costs 221 s against a 150 s baseline. The family is the variable: at the same 100 requests, SQL injection 147 s with six declared sinks, path 194 s with sixteen, output encoding 704 s with seven. And the driver named its own expensive step once it was asked to: 2,435 evidence requests in 45.5 s, then 315 arbitration requests unanswered at 1,200 s, then a split abandoned for want of time. It survived since September because every reproduction outside the driver carried the SQL sinks -- the family the declared target belongs to, and the one family that is cheap on this subject. Nine hypotheses were eliminated by re-measuring the cheap case. Splitting never rescued it because a failed batch is split by REQUEST COUNT and each part carries the same everywhere-sinks. contributor-scan 7.7 carries the fix: budget a batch by the sink nodes it implies, using the count the evidence pass already computes, and report a family whose sinks are everywhere as expensive rather than as failed. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/specs/contributor-scan/tasks.md | 9 ++ .kiro/specs/pre-push-safety-net/tasks.md | 7 +- .../2026-09-22-php-query-cost-identified.json | 121 ++++++++++++++++++ 3 files changed, 136 insertions(+), 1 deletion(-) create mode 100644 benchmarks/measurements/2026-09-22-php-query-cost-identified.json diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 0ded40c..9c9f75a 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -1262,3 +1262,12 @@ the ontology describes libraries, frameworks and language constructs, never repo - Done when a deliberate change to a matcher or a fact table fails the diff, and the expectation records the ruleset and engine identities it was taken under. - _Boundary: evaluation harness_ - _Requirements: 8.1, 8.2_ + +- [ ] 7.7 Budget a batch by the sinks it implies, not by the requests it holds + - **Named by the 2026-09-22 identification.** A taint batch costs what its matched sink NODES cost, because `reachableByFlows` is asked per sink. Measured at a fixed 100 requests on one graph: SQL injection 147 s, path 194 s, output encoding 704 s — and output encoding declares seven sink names against path's sixteen. What separates them is that `echo` and `print` match thousands of calls in a WordPress plugin. + - The driver currently splits a failed batch into eight parts by request count, which cannot help: each part carries the same everywhere-sinks. + - The count needed is already computed. `taint.sc` memoises its candidate sink list per sink set, and the evidence pass reports the in-scope sink list per request before any dataflow runs. A driver that reads it can size a batch by expected work, and can report a family whose sinks are everywhere as EXPENSIVE rather than as failed. + - Done when a batch is sized by the sink nodes it implies, the measured plugin's arbitration completes within its ceiling, and a family too expensive to arbitrate is reported as such rather than as a timeout. + - _Boundary: contributor-scan scan budget, cpg backend_ + - _Depends: 7.3_ + - _Requirements: 4.1, 6.3, 8.1_ diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index 35554cd..d2dbba0 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -465,7 +465,12 @@ Preserve all completed task evidence below. - _The scale limit, quantified._ Roughly one to two seconds per question. 25 and 100 regions complete cleanly; 500 regions asks 2,526 questions, needs more than the 1,800-second ceiling and is killed. Nothing pathological, and the retry's fast failure is still unexplained. - _What is NOT the problem, each measured rather than assumed:_ the malformed-CPG bug (fixed in 4.0.625, verified on the real trigger file); the dataflow overlay (42 s on the full 637-file graph); the engine version (4.0.625 and 4.0.631 indistinguishable); and the bundled Stripe, Braintree and Twocheckout SDKs under `includes/lib`, which the vendored list does not cover — that was the hypothesis and the overlay costs 42 s with them and 37 s without, so it is refuted. - **The remaining real gap is bookkeeping, not detection:** `questions_completed` is zero in every PHP run including the ones that find things, with `cross_partition_semantics_unresolved` and `partition_file_census_incomplete` present throughout. Findings and question outcomes disagree in public, and the admission policy reads the outcomes. Group 12 raised completion for JavaScript and Python and PHP was never measured. Evidence: `benchmarks/measurements/2026-09-20-php-detection-corrected.json`. - - **Blocker still unidentified; investigation paused 2026-09-19 to return to vulnerability work.** Six hypotheses were measured and eliminated: batch size (17,700 requests answer in 44 s with a 2.6 MB response), dataflow cost (50 execution requests in 24 s), call depth 3 with parameter sources (24.7 s for 20), argument length (a 16 KB hook table against a 128 KB cap), hook callback resolution (1 s difference with 233 hooks), and build or sharding (79 s, no shards). Preparation measures 273.5 s. Every reproduction of the failing 22,125-request batch outside the push path answers in under a minute, and the difference from the production invocation is not known. The exception has never been read, because the backend logged a stack trace's tail and the corrected extractor reports its head, which on this engine is JVM startup noise. Evidence: `benchmarks/measurements/2026-09-19-php-blocker-open.json`. + - **IDENTIFIED 2026-09-22.** The cost is the number of CALL NODES a family's sink names match in the repository. Not the request count, not the declared sink count, not any request parameter. `reachableByFlows` is asked per matched sink, and PHP's output-encoding sinks are `echo` and `print`, which a WordPress plugin writes thousands of times. + - _Three controlled sweeps on one graph, built once._ Batch size is exonerated: 400 requests answer in 170 s and cost per request FALLS with size, because roughly 70 s is fixed JVM start and graph load. Request parameters are exonerated: at 200 requests the full production shape, call depth 3 with parameter sources, costs 221 s against a 150 s baseline. The family is the variable: at the same 100 requests, SQL injection costs 147 s with six declared sinks, path costs 194 s with sixteen, and output encoding costs 704 s with seven. + - _And the driver named its own expensive step, once it was asked to._ `taint: 2435 request(s) in 45.5s, answered` for the evidence pass, then `taint: 315 request(s) in 1200.1s, unanswered` for arbitration across all families, then a split abandoned for want of time. + - _Why it survived since September:_ every reproduction outside the driver carried the SQL sinks, because that is the family the declared target belongs to. The one family that is cheap on this subject was the one being re-measured, and nine hypotheses were eliminated that way. + - _Why splitting never rescued it:_ a failed batch is split into eight parts BY REQUEST COUNT, and each part carries the same everywhere-sinks, so the whole is no cheaper. Splitting by request count cannot help when the cost is per matched sink. Evidence: `benchmarks/measurements/2026-09-22-php-query-cost-identified.json`. + - **Blocker identified; the fix is contributor-scan 7.7.** Six hypotheses were measured and eliminated: batch size (17,700 requests answer in 44 s with a 2.6 MB response), dataflow cost (50 execution requests in 24 s), call depth 3 with parameter sources (24.7 s for 20), argument length (a 16 KB hook table against a 128 KB cap), hook callback resolution (1 s difference with 233 hooks), and build or sharding (79 s, no shards). Preparation measures 273.5 s. Every reproduction of the failing 22,125-request batch outside the push path answers in under a minute, and the difference from the production invocation is not known. The exception has never been read, because the backend logged a stack trace's tail and the corrected extractor reports its head, which on this engine is JVM startup noise. Evidence: `benchmarks/measurements/2026-09-19-php-blocker-open.json`. - _Boundary: Evaluation harness, full-run integration_ - _Depends: 13.1_ - _Requirements: 6.2, 6.3, 8.1, 8.2_ diff --git a/benchmarks/measurements/2026-09-22-php-query-cost-identified.json b/benchmarks/measurements/2026-09-22-php-query-cost-identified.json new file mode 100644 index 0000000..2ef3b9b --- /dev/null +++ b/benchmarks/measurements/2026-09-22-php-query-cost-identified.json @@ -0,0 +1,121 @@ +{ + "schema_version": 1, + "experiment": "php-query-cost-identified-v1", + "date": "2026-09-22", + "task": "pre-push-safety-net 13.2", + "claim_scope": "What makes a PHP arbitration batch unaffordable. One graph, built once, three controlled sweeps plus the driver's own log.", + "the_answer": "The cost is the number of CALL NODES a family's sink names match in this repository, not the number of requests in the batch, not the number of sink names declared, and not any request parameter. `reachableByFlows` is asked per matched sink, and PHP's output-encoding sinks are `echo` and `print`, which a WordPress plugin writes thousands of times.", + "the_evidence": { + "batch_size_is_exonerated": { + "sweep": [ + { + "requests": 25, + "seconds": 71.4, + "answered": true, + "rc": 0, + "per_request": 2.86 + }, + { + "requests": 50, + "seconds": 76.0, + "answered": true, + "rc": 0, + "per_request": 1.52 + }, + { + "requests": 100, + "seconds": 155.4, + "answered": true, + "rc": 0, + "per_request": 1.55 + }, + { + "requests": 200, + "seconds": 166.7, + "answered": true, + "rc": 0, + "per_request": 0.83 + }, + { + "requests": 400, + "seconds": 169.7, + "answered": true, + "rc": 0, + "per_request": 0.42 + } + ], + "reading": "400 requests answer in 170 s and cost per request FALLS with size, because roughly 70 s is fixed JVM start and graph load." + }, + "request_parameters_are_exonerated": { + "sweep": [ + { + "callDepth": "0", + "parameterSources": "false", + "seconds": 150.2, + "answered": true, + "rc": 0 + }, + { + "callDepth": "3", + "parameterSources": "false", + "seconds": 156.9, + "answered": true, + "rc": 0 + }, + { + "callDepth": "0", + "parameterSources": "true", + "seconds": 226.0, + "answered": true, + "rc": 0 + }, + { + "callDepth": "3", + "parameterSources": "true", + "seconds": 221.3, + "answered": true, + "rc": 0 + } + ], + "reading": "At 200 requests the full production shape, call depth 3 with parameter sources, costs 221 s against a 150 s baseline. Not the difference between 170 s and a 1,200 s timeout." + }, + "the_family_is_the_variable": { + "sweep": [ + { + "family": "injection_sql", + "sinks_declared": 6, + "seconds": 146.8, + "answered": true + }, + { + "family": "output_encoding", + "sinks_declared": 7, + "seconds": 704.2, + "answered": true + }, + { + "family": "path", + "sinks_declared": 16, + "seconds": 193.8, + "answered": true + } + ], + "reading": "Same graph, same 100 requests, same query. SQL injection costs 147 s with six declared sinks; path costs 194 s with sixteen; output encoding costs 704 s with seven. The declared count does not predict it and the request count does not predict it. What separates output encoding is that `echo` and `print` match thousands of calls where `$wpdb->get_var` matches a handful." + }, + "the_driver_named_its_own_expensive_step": [ + "cpg batch taint: 2435 request(s) in 45.5s, answered (evidence, no dataflow)", + "cpg batch taint: 65 request(s) in 21.6s, answered", + "cpg batch taint: 315 request(s) in 1200.1s, unanswered (arbitration, all families)", + "cpg batch taint: split abandoned at part 1, no time left", + "cpg batch dominance: 13 request(s) in 20.6s, answered" + ] + }, + "why_it_stayed_unidentified_since_september": "Every reproduction outside the driver carried the SQL sinks, because that is the family the declared target belongs to. The one family that is cheap on this subject was the one being re-measured. Nine hypotheses were eliminated that way, and the tenth was not a hypothesis anyone had stated.", + "why_splitting_did_not_rescue_it": "The driver splits a failed batch into eight parts BY REQUEST COUNT. Each part still carries the same everywhere-sinks, so each part is expensive in proportion to its share and the whole is no cheaper. Splitting by request count cannot help when the cost is per matched sink.", + "fix_direction": "Budget a batch by the sink nodes it implies rather than by the requests it holds. The count is already computed: taint.sc memoises its candidate sink list per sink set, and the evidence pass reports the in-scope sink list per request before any dataflow runs. A driver that reads it can size batches, and can report a family whose sinks are everywhere as expensive rather than as failed.", + "limits": [ + "One subject and one language. The effect should be proportional to how common a family's sinks are in a codebase, which is a property of the code.", + "The sweeps use file-scoped requests; production also sends function-scoped ones, which were not separately timed.", + "`echo` as an output-encoding sink is correct for PHP. The finding is about cost, not about whether the fact is right." + ] +}