From 537483657d72dfca09407843dd4dabcac7ee7046 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Fri, 18 Sep 2026 13:22:05 +0200 Subject: [PATCH] test(m3): PHP still not established; blocker is the dataflow execution pass Retried the PMPro transfer with batch splitting, a 4500-second deadline and a 900-second query ceiling. Stopped after 26 minutes with no case completed and no verdict, so PHP transfer remains unestablished. A probe narrowed the cause and corrected my earlier reading of it. The evidence pass answers all 4,425 PMPro PHP regions in ONE batch in 29.7 seconds with a 681 KB response, exit 0, and its cost is fixed rather than per request: one request took 40.2 s and 4,425 took 29.7 s. Neither request count nor response size is anywhere near a limit, so batch size and serialization are not the cause as I first supposed from a single extrapolated data point. What fails is the execution pass, which runs reachableByFlows per candidate. The probe's own scope is recorded with it: it sent evidenceOnly=true and therefore measured the structural pass, not the failing one. Those numbers would otherwise read as evidence that PMPro's taint work is cheap, which they are not. Splitting stays, because it is correct on its own terms and recovers a batch that fails whole. It is worth noting it also makes an unrecoverable batch slower to abandon, since each of the eight parts burns its own share of the query ceiling. M3 remains two of three: Node established in M1b, Python established, PHP blocked on dataflow cost for a 1,274-file repository. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .../pre-push-safety-net/release-milestones.md | 14 ++-- .kiro/specs/pre-push-safety-net/tasks.md | 2 + .../2026-09-17-m3-php-transfer.json | 21 +++++- .../2026-09-18-php-batch-diagnosis.json | 72 +++++++++++++++++++ 4 files changed, 102 insertions(+), 7 deletions(-) create mode 100644 benchmarks/measurements/2026-09-18-php-batch-diagnosis.json diff --git a/.kiro/specs/pre-push-safety-net/release-milestones.md b/.kiro/specs/pre-push-safety-net/release-milestones.md index a8cec6c..c44a34c 100644 --- a/.kiro/specs/pre-push-safety-net/release-milestones.md +++ b/.kiro/specs/pre-push-safety-net/release-milestones.md @@ -144,11 +144,15 @@ zero admitted defects. That includes the SQL injection this spec names as previo and both broken-object-level-authorization targets, which is the first time access control has answered on Python at all. -PHP: not established. All three PMPro comparisons exhausted a 3000-second deadline with a -1200-second query ceiling and produced no artifact, so there is no target evidence. The taint -batch fails on the 1,274-file tree, first with an engine exception and then by timeout. The pins, -receipts and declared target are correct, so this is a runtime blocker on a large repository -rather than evidence about whether the design transfers. +PHP: not established, and the blocker is now named. All PMPro comparisons exhausted every budget +tried, up to a 4500-second deadline with a 900-second query ceiling, and produced no artifact, so +there is no target evidence. Splitting a failed batch into parts did not rescue it. + +A probe narrowed the cause: the evidence pass answers all 4,425 PMPro regions in one batch in +29.7 seconds with a 681 KB response, and its cost is fixed rather than per request. Neither +request count nor response size is anywhere near a limit. What fails is the execution pass, which +runs `reachableByFlows` per candidate. The pins, receipts and declared target are correct, so the +question is being asked properly and the dataflow work is what cannot finish. The exit asks for Node, PHP and Python together. Two of three are in hand. diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index 0a8573d..519bbc6 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -456,6 +456,8 @@ Preserve all completed task evidence below. - Done when all three comparisons complete with their declared target evidence recorded, or the budget they need is stated as a finding rather than reported as a pass. - _Measured 2026-09-17, PHP transfer NOT established._ All three comparisons exhausted a 3000-second deadline with a 1200-second query ceiling and produced no artifact, so there is no target evidence at all. The engine reports `cpg batch taint failed` twice per case on the 1,274-file PMPro tree: first an exception whose trace runs through ujson array transformation, then a timeout on the retry. Budgets tried: 900 s, 1200 s and 3000 s deadlines against 300 s and 1200 s query ceilings. Pins, source receipts and the declared target are all correct, so the question was asked properly and the engine could not answer it. This is a runtime blocker on a large repository, not evidence about whether the design transfers to PHP. Evidence: `benchmarks/measurements/2026-09-17-m3-php-transfer.json`. - _Two harness defects found and fixed on the way, each of which had made an earlier run look like a transfer failure:_ the manifest labels a case by what it is while the id says what the comparison does, and filtering on one while keying expectations by the other ran a single case; and the per-query ceiling is taken as the minimum with the remaining transaction time, so raising only the deadline could never have helped. + - _Retried 2026-09-18 with batch splitting, still not established._ A batch that fails whole is now retried in eight parts, which a probe made look promising: the evidence pass answers all 4,425 PMPro regions in one batch in 29.7 s with a 681 KB response, and cost is fixed rather than per request, so neither count nor size is near a limit. That probe sent `evidenceOnly=true` and therefore measured the structural pass, not the failing one; recorded because those numbers would otherwise read as evidence that PMPro's taint work is cheap. With a 4500-second deadline and a 900-second query ceiling the run was stopped after 26 minutes with no case completed. Splitting also makes an unrecoverable batch slower to abandon, since each part burns its own share of the ceiling. + - **Blocker named: the execution pass, which runs `reachableByFlows` per candidate.** Not batch size, not serialization, not the deadline. Evidence: `benchmarks/measurements/2026-09-18-php-batch-diagnosis.json`. - _Boundary: Evaluation harness, full-run integration_ - _Depends: 13.1_ - _Requirements: 6.2, 6.3, 8.1, 8.2_ diff --git a/benchmarks/measurements/2026-09-17-m3-php-transfer.json b/benchmarks/measurements/2026-09-17-m3-php-transfer.json index c43d4a6..fe631cc 100644 --- a/benchmarks/measurements/2026-09-17-m3-php-transfer.json +++ b/benchmarks/measurements/2026-09-17-m3-php-transfer.json @@ -140,7 +140,7 @@ "sha256": "6753e7e53695016da04f0092ad54fea5c5fc7470d4de0c5e284721d68ca22337", "files": 10 }, - "outcome": "PHP TRANSFER NOT ESTABLISHED", + "outcome": "PHP TRANSFER NOT ESTABLISHED; blocker narrowed to the dataflow execution pass", "what_failed": "All three comparisons exhausted a 3000-second deadline and produced no artifact, so there is no target evidence at all. The engine reports `cpg batch taint failed` twice per case on the 1,274-file PMPro tree: first with an exception whose trace runs through ujson array transformation, then by timeout on the retry.", "budgets_tried": [ { @@ -166,5 +166,22 @@ "reading": "This is a runtime blocker on a large repository, not evidence about whether the design transfers to PHP. The pins, source receipts and declared target are all correct, so the question was asked properly and the engine could not answer it within any budget tried. PHP transfer stays unestablished and M3 stays unmet.", "limits": [ "No detection, precision or admission claim for PHP: nothing completed to claim anything about." - ] + ], + "retry_with_batch_splitting": { + "date": "2026-09-18", + "change": "A batch that fails whole is retried in eight parts; each request is independent.", + "budgets": { + "deadline_seconds": 4500, + "query_timeout_seconds": 900 + }, + "result": "stopped by the maintainer after 26 minutes with no case completed and no verdict produced", + "reading": "Splitting did not rescue the execution pass within a time anyone would wait. It also makes an unrecoverable batch slower to give up on, because each of the eight parts burns its own share of the query ceiling before the next is tried, so one timeout becomes several." + }, + "narrowed_by_probe": { + "evidence_pass": "All 4,425 PMPro PHP regions answer in ONE batch in 29.7 s with a 681 KB response, exit 0.", + "cost_shape": "Fixed, not per request: 1 request took 40.2 s and 4,425 took 29.7 s.", + "conclusion": "Neither request count nor response size is near a limit. The blocker is the execution pass, which runs reachableByFlows per candidate, and not batch size or serialization as first supposed.", + "instrument_note": "That probe sent evidenceOnly=true, so it measured the structural pass and not the failing one. Recorded because those numbers would otherwise read as evidence that PMPro's taint work is cheap.", + "evidence": "benchmarks/measurements/2026-09-18-php-batch-diagnosis.json" + } } diff --git a/benchmarks/measurements/2026-09-18-php-batch-diagnosis.json b/benchmarks/measurements/2026-09-18-php-batch-diagnosis.json new file mode 100644 index 0000000..9bbad68 --- /dev/null +++ b/benchmarks/measurements/2026-09-18-php-batch-diagnosis.json @@ -0,0 +1,72 @@ +{ + "experiment": "php-taint-batch-v1", + "root": "/cache/pmpro/1a900be3f09f", + "attempts": [ + { + "requests": 1, + "seconds": 40.16467795101926, + "exit": 0, + "stdout_bytes": 36400, + "answered": true, + "answer_keys": 2, + "stderr_head": "Picked up JAVA_TOOL_OPTIONS: -Xmx2048m\nPicked up JAVA_TOOL_OPTIONS: -Xmx2048m\nexecuting /app/src/openultrasast/cpg/queries/taint.sc with params=Map(cpgFile -> /results/php-probe/cpg.bin, requestsFile -> /results/php-probe/requests-1.json)\nCreating project `cpg.bin` for CPG at `/results/php-probe/cpg.bin`\nCreating working copy of CPG to be safe\nLoading base CPG from: /results/php-probe/workspace/cpg.bin/cpg.bin.tmp\nOverlay dataflowOss already exists - skipping\nThe graph has been modified. You may want to use the `save` command to persist changes to disk. All changes will also be saved collectively on exit\nclosing/saving project `cpg.bin`\n", + "stderr_bytes": 646 + }, + { + "requests": 10, + "seconds": 39.16813864791766, + "exit": 0, + "stdout_bytes": 37689, + "answered": true, + "answer_keys": 11, + "stderr_head": "Picked up JAVA_TOOL_OPTIONS: -Xmx2048m\nPicked up JAVA_TOOL_OPTIONS: -Xmx2048m\nexecuting /app/src/openultrasast/cpg/queries/taint.sc with params=Map(cpgFile -> /results/php-probe/cpg.bin, requestsFile -> /results/php-probe/requests-10.json)\nCreating project `cpg.bin` for CPG at `/results/php-probe/cpg.bin`\nProject with name cpg.bin already exists - overwriting\nCreating working copy of CPG to be safe\nLoading base CPG from: /results/php-probe/workspace/cpg.bin/cpg.bin.tmp\nOverlay dataflowOss already exists - skipping\nThe graph has been modified. You may want to use the `save` command to persist changes to disk. All changes will also be saved collectively on exit\nclosing/saving project `cpg.bin`\n", + "stderr_bytes": 702 + }, + { + "requests": 50, + "seconds": 30.821678715990856, + "exit": 0, + "stdout_bytes": 43445, + "answered": true, + "answer_keys": 51, + "stderr_head": "Picked up JAVA_TOOL_OPTIONS: -Xmx2048m\nPicked up JAVA_TOOL_OPTIONS: -Xmx2048m\nexecuting /app/src/openultrasast/cpg/queries/taint.sc with params=Map(cpgFile -> /results/php-probe/cpg.bin, requestsFile -> /results/php-probe/requests-50.json)\nCreating project `cpg.bin` for CPG at `/results/php-probe/cpg.bin`\nProject with name cpg.bin already exists - overwriting\nCreating working copy of CPG to be safe\nLoading base CPG from: /results/php-probe/workspace/cpg.bin/cpg.bin.tmp\nOverlay dataflowOss already exists - skipping\nThe graph has been modified. You may want to use the `save` command to persist changes to disk. All changes will also be saved collectively on exit\nclosing/saving project `cpg.bin`\n", + "stderr_bytes": 702 + }, + { + "requests": 200, + "seconds": 31.992039224831387, + "exit": 0, + "stdout_bytes": 65147, + "answered": true, + "answer_keys": 201, + "stderr_head": "Picked up JAVA_TOOL_OPTIONS: -Xmx2048m\nPicked up JAVA_TOOL_OPTIONS: -Xmx2048m\nexecuting /app/src/openultrasast/cpg/queries/taint.sc with params=Map(cpgFile -> /results/php-probe/cpg.bin, requestsFile -> /results/php-probe/requests-200.json)\nCreating project `cpg.bin` for CPG at `/results/php-probe/cpg.bin`\nProject with name cpg.bin already exists - overwriting\nCreating working copy of CPG to be safe\nLoading base CPG from: /results/php-probe/workspace/cpg.bin/cpg.bin.tmp\nOverlay dataflowOss already exists - skipping\nThe graph has been modified. You may want to use the `save` command to persist changes to disk. All changes will also be saved collectively on exit\nclosing/saving project `cpg.bin`\n", + "stderr_bytes": 703 + }, + { + "requests": 4425, + "seconds": 29.67125784815289, + "exit": 0, + "stdout_bytes": 681199, + "answered": true, + "answer_keys": 4426, + "stderr_head": "Picked up JAVA_TOOL_OPTIONS: -Xmx2048m\nPicked up JAVA_TOOL_OPTIONS: -Xmx2048m\nexecuting /app/src/openultrasast/cpg/queries/taint.sc with params=Map(cpgFile -> /results/php-probe/cpg.bin, requestsFile -> /results/php-probe/requests-4425.json)\nCreating project `cpg.bin` for CPG at `/results/php-probe/cpg.bin`\nProject with name cpg.bin already exists - overwriting\nCreating working copy of CPG to be safe\nLoading base CPG from: /results/php-probe/workspace/cpg.bin/cpg.bin.tmp\nOverlay dataflowOss already exists - skipping\nThe graph has been modified. You may want to use the `save` command to persist changes to disk. All changes will also be saved collectively on exit\nclosing/saving project `cpg.bin`\n", + "stderr_bytes": 704 + } + ], + "build": { + "seconds": 51.444085616152734, + "exit": 0, + "bytes": 12062695 + }, + "regions": { + "total": 4461, + "php": 4425 + }, + "milestone": "M3 PHP diagnosis", + "instrument_scope": "This probe sent evidenceOnly=true, so it measured the EVIDENCE pass, which is structural and does no dataflow. That is not the pass that fails in production. Stating it plainly because the numbers would otherwise read as evidence that PMPro's taint work is cheap, which they are not.", + "what_it_establishes": "The evidence pass scales on PMPro without difficulty: all 4,425 PHP regions answered in one batch in 29.7 seconds with a 681 KB response, exit 0. Cost is fixed overhead, not per request: one request took 40.2 s and 4,425 took 29.7 s. So neither request count nor response size is near any limit in that mode.", + "what_it_narrows": "The production failure is therefore specific to the EXECUTION pass, which runs reachableByFlows per candidate. Batch size and serialization are not the cause on their own; the dataflow work is.", + "limits": [ + "Evidence mode only, one PHP repository, one machine. No detection or latency claim." + ] +}