From 73e9c1fed69986eb8c23f25d2888ad68a0eb0c08 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 21 Sep 2026 10:57:19 +0200 Subject: [PATCH] measure(m3): two languages in a repository defeat completion, and so defeat admission No PHP question is ever marked completed, and PHP was never the variable. partitions.py raises cross_partition_semantics_unresolved whenever more than one language partition exists, that reason is a member of GRAPH_INTEGRITY_GAPS, and a graph-integrity gap demotes every completed outcome in the scan. Admission requires a completed outcome. The evidence is one column wide. NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat own degradation, vendor_semantics_unresolved, is a DECLARED exclusion rather than an integrity gap, which is exactly why it does not demote. This lands on the declared v0.1 target. A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it, which is why a census across a third subject was worth running. Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task 12.3 already fixed this bug class one level down by making a gap question-owned so it could not block another question comparison; the same principle applies to a partition boundary. Reclassifying the reason as a declared exclusion is explicitly the wrong fix, because that would claim cross-language flows are out of scope when they are unresolved. Task 13.4 rewritten with the diagnosis, and the milestone amendment now ranks it with M2 rather than behind it: a narrow JavaScript-only release can proceed without it, the WordPress target cannot proceed at all. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .../pre-push-safety-net/release-milestones.md | 10 +++ .kiro/specs/pre-push-safety-net/tasks.md | 11 +-- ...09-21-two-languages-defeat-completion.json | 69 +++++++++++++++++++ 3 files changed, 86 insertions(+), 4 deletions(-) create mode 100644 benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json diff --git a/.kiro/specs/pre-push-safety-net/release-milestones.md b/.kiro/specs/pre-push-safety-net/release-milestones.md index d4295d0..8c71510 100644 --- a/.kiro/specs/pre-push-safety-net/release-milestones.md +++ b/.kiro/specs/pre-push-safety-net/release-milestones.md @@ -48,6 +48,16 @@ threshold and no owner boundary. it would ship silence. Output encoding, path, prototype and deserialization have not earned a place either. M4 explicitly does not require every language or family to become eligible. +**Amendment to the amendment, same day.** Decision 1's re-run exposed something larger than the +runtime limit it was chasing. A repository holding two languages can never complete a question, +because a partition boundary is classified as a graph-integrity gap and such a gap demotes every +completed outcome; and a completed outcome is what admission requires. NodeGoat completes 299 of +299 and VAmPI 35 of 35 because each holds one language, while PMPro completes none of 520 and +still establishes 43 findings. The declared v0.1 target is WordPress plugins, and a plugin is PHP +plus its admin JavaScript, so on the target this tool was built for it can detect and never admit. +Task 13.4 carries it, and it now ranks with M2 rather than behind it: a narrow JavaScript-only +release can proceed without it, and the WordPress target cannot proceed at all. + **The risk this amendment does not remove.** Every number so far comes from two repositories used to build the analyser, and one week of examining them surfaced seven false positives. On untouched code the precision estimate will move, against a bar of 95% with zero false alerts on the fixed and diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index 7fceb14..0bac5e0 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -561,10 +561,13 @@ Preserve all completed task evidence below. - _Boundary: Evaluation harness, full-run integration_ - _Depends: 13.2_ - _Requirements: 6.2, 6.3, 8.1, 8.2_ -- [ ] 13.4 Complete a PHP question, or say which scope it could not reach - - **Named by the 2026-09-20 correction.** `questions_completed` is zero in every PHP run, including the runs that establish findings, with `cross_partition_semantics_unresolved` and `partition_file_census_incomplete` present throughout. Findings and question outcomes therefore disagree in public, and the admission policy reads the outcomes rather than the findings. - - Group 12 raised completion from 205 to 298 of 436 for JavaScript and answered every VAmPI question; PHP was never measured. The partition and shard boundaries are the first place to look, because those are the degradations every run carries. - - Done when a PHP question whose scope the graph holds completes, a question whose scope it cannot reach stays unresolved with the reason named, and no question completes by widening its scope. +- [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds + - **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan. + - The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote. + - **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it. + - Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task 12.3 already fixed this bug class one level down, making an empty-scope gap question-owned "so it cannot block another question's comparison"; the same principle applies to a partition boundary. + - Do not reclassify the reason as a declared exclusion. That would claim cross-language flows are out of scope when they are unresolved, which is the opposite of what this project's degradation vocabulary exists to say. + - Done when a question whose scope lies inside one partition completes, a question whose scope could cross a partition boundary stays unresolved with the boundary named, and no question completes by widening its scope. Evidence: `benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json`. - _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_ - _Depends: 13.3_ - _Requirements: 2.1, 2.2, 6.3, 8.1_ diff --git a/benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json b/benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json new file mode 100644 index 0000000..84ce354 --- /dev/null +++ b/benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json @@ -0,0 +1,69 @@ +{ + "schema_version": 1, + "experiment": "two-languages-defeat-completion-v1", + "date": "2026-09-21", + "task": "pre-push-safety-net 13.4", + "claim_scope": "Why no PHP question is ever marked completed. Measured across four census runs on three subjects.", + "finding": "A repository that holds TWO languages can never complete a question, so it can never admit a finding. The cause is structural and deliberate, not a bug in PHP support: `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists, that reason is a member of `GRAPH_INTEGRITY_GAPS`, and a graph-integrity gap demotes EVERY completed outcome in the scan. PHP was never the variable.", + "evidence": { + "nodegoat": { + "languages": { + "javascript": 44 + }, + "language_count": 1, + "questions_total": 299, + "questions_completed": 299, + "findings": 8, + "degradations": [ + "vendor_semantics_unresolved" + ] + }, + "vampi": { + "languages": { + "python": 25 + }, + "language_count": 1, + "questions_total": 35, + "questions_completed": 35, + "findings": 4, + "degradations": [] + }, + "pmpro-n25": { + "languages": { + "php": 4425, + "javascript": 36 + }, + "language_count": 2, + "questions_total": 127, + "questions_completed": 0, + "findings": 19, + "degradations": [ + "cross_partition_semantics_unresolved", + "partition_file_census_incomplete" + ] + }, + "pmpro-n100": { + "languages": { + "php": 4425, + "javascript": 36 + }, + "language_count": 2, + "questions_total": 520, + "questions_completed": 0, + "findings": 43, + "degradations": [ + "cross_partition_semantics_unresolved", + "partition_file_census_incomplete" + ] + } + }, + "the_reading": "NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520, while establishing 19 and 43 findings. The only difference between the subjects that complete and the one that does not is the number of languages. NodeGoat's own degradation, `vendor_semantics_unresolved`, is a DECLARED exclusion rather than an integrity gap, which is exactly why it does not demote.", + "why_this_matters_for_the_release": "The declared v0.1 target is WordPress plugins, and a WordPress plugin is PHP plus its admin JavaScript. Every such repository is multi-language, so on the declared target the analyser can detect defects and can never admit one. That is a larger obstacle than the PHP runtime limit, and it is invisible from either subject that currently passes.", + "why_the_gap_exists": "A flow that crosses a language boundary is not modelled: PHP calling into JavaScript, or a script posting to an endpoint, is semantics the partitioned graph cannot follow. Recording that is right.", + "why_the_response_is_too_coarse": "The gap demotes every question, including one wholly inside a single PHP file with no cross-language element in its scope. This project has already fixed this bug class once, at the question level: task 12.3 made an empty-scope gap question-owned 'so it cannot block another question's comparison'. The same principle applies one level up. A partition boundary should demote the questions whose scope could cross it, not every question in the repository.", + "what_to_do": "Scope the partition boundary to the questions it can actually affect, and leave a question whose scope lies inside one partition completable. Do not simply reclassify the reason as a declared exclusion: that would silently claim cross-language flows are out of scope rather than unresolved.", + "limits": [ + "Three subjects, one of them multi-language. A single-language PHP repository has not been measured and would isolate the cause further.", + "`partition_file_census_incomplete` is present on PMPro too and is also an integrity gap, so fixing only the cross-partition reason may not be enough on this subject." + ] +}