measure(m3): two languages in a repository defeat completion, and so defeat admission

No PHP question is ever marked completed, and PHP was never the variable. partitions.py raises
cross_partition_semantics_unresolved whenever more than one language partition exists, that reason
is a member of GRAPH_INTEGRITY_GAPS, and a graph-integrity gap demotes every completed outcome in
the scan. Admission requires a completed outcome.

The evidence is one column wide. NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is
Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and
completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat own degradation,
vendor_semantics_unresolved, is a DECLARED exclusion rather than an integrity gap, which is exactly
why it does not demote.

This lands on the declared v0.1 target. A WordPress plugin is PHP plus its admin JavaScript, so
every one of them is multi-language, so on the target this tool was built for it can detect defects
and never admit one. Neither subject that currently passes could reveal it, which is why a census
across a third subject was worth running.

Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph
cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task
12.3 already fixed this bug class one level down by making a gap question-owned so it could not
block another question comparison; the same principle applies to a partition boundary. Reclassifying
the reason as a declared exclusion is explicitly the wrong fix, because that would claim
cross-language flows are out of scope when they are unresolved.

Task 13.4 rewritten with the diagnosis, and the milestone amendment now ranks it with M2 rather
than behind it: a narrow JavaScript-only release can proceed without it, the WordPress target
cannot proceed at all.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-21 10:57:19 +02:00
co-authored by Claude Opus 5
parent 6bd6e3531e
commit 73e9c1fed6
3 changed files with 86 additions and 4 deletions
@@ -48,6 +48,16 @@ threshold and no owner boundary.
it would ship silence. Output encoding, path, prototype and deserialization have not earned a it would ship silence. Output encoding, path, prototype and deserialization have not earned a
place either. M4 explicitly does not require every language or family to become eligible. place either. M4 explicitly does not require every language or family to become eligible.
**Amendment to the amendment, same day.** Decision 1's re-run exposed something larger than the
runtime limit it was chasing. A repository holding two languages can never complete a question,
because a partition boundary is classified as a graph-integrity gap and such a gap demotes every
completed outcome; and a completed outcome is what admission requires. NodeGoat completes 299 of
299 and VAmPI 35 of 35 because each holds one language, while PMPro completes none of 520 and
still establishes 43 findings. The declared v0.1 target is WordPress plugins, and a plugin is PHP
plus its admin JavaScript, so on the target this tool was built for it can detect and never admit.
Task 13.4 carries it, and it now ranks with M2 rather than behind it: a narrow JavaScript-only
release can proceed without it, and the WordPress target cannot proceed at all.
**The risk this amendment does not remove.** Every number so far comes from two repositories used **The risk this amendment does not remove.** Every number so far comes from two repositories used
to build the analyser, and one week of examining them surfaced seven false positives. On untouched to build the analyser, and one week of examining them surfaced seven false positives. On untouched
code the precision estimate will move, against a bar of 95% with zero false alerts on the fixed and code the precision estimate will move, against a bar of 95% with zero false alerts on the fixed and
+7 -4
View File
@@ -561,10 +561,13 @@ Preserve all completed task evidence below.
- _Boundary: Evaluation harness, full-run integration_ - _Boundary: Evaluation harness, full-run integration_
- _Depends: 13.2_ - _Depends: 13.2_
- _Requirements: 6.2, 6.3, 8.1, 8.2_ - _Requirements: 6.2, 6.3, 8.1, 8.2_
- [ ] 13.4 Complete a PHP question, or say which scope it could not reach - [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds
- **Named by the 2026-09-20 correction.** `questions_completed` is zero in every PHP run, including the runs that establish findings, with `cross_partition_semantics_unresolved` and `partition_file_census_incomplete` present throughout. Findings and question outcomes therefore disagree in public, and the admission policy reads the outcomes rather than the findings. - **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan.
- Group 12 raised completion from 205 to 298 of 436 for JavaScript and answered every VAmPI question; PHP was never measured. The partition and shard boundaries are the first place to look, because those are the degradations every run carries. - The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote.
- Done when a PHP question whose scope the graph holds completes, a question whose scope it cannot reach stays unresolved with the reason named, and no question completes by widening its scope. - **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it.
- Recording the boundary is right: a flow from PHP into JavaScript is semantics the partitioned graph cannot follow. Demoting every question is not, including one wholly inside a single PHP file. Task 12.3 already fixed this bug class one level down, making an empty-scope gap question-owned "so it cannot block another question's comparison"; the same principle applies to a partition boundary.
- Do not reclassify the reason as a declared exclusion. That would claim cross-language flows are out of scope when they are unresolved, which is the opposite of what this project's degradation vocabulary exists to say.
- Done when a question whose scope lies inside one partition completes, a question whose scope could cross a partition boundary stays unresolved with the boundary named, and no question completes by widening its scope. Evidence: `benchmarks/measurements/2026-09-21-two-languages-defeat-completion.json`.
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_ - _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
- _Depends: 13.3_ - _Depends: 13.3_
- _Requirements: 2.1, 2.2, 6.3, 8.1_ - _Requirements: 2.1, 2.2, 6.3, 8.1_
@@ -0,0 +1,69 @@
{
"schema_version": 1,
"experiment": "two-languages-defeat-completion-v1",
"date": "2026-09-21",
"task": "pre-push-safety-net 13.4",
"claim_scope": "Why no PHP question is ever marked completed. Measured across four census runs on three subjects.",
"finding": "A repository that holds TWO languages can never complete a question, so it can never admit a finding. The cause is structural and deliberate, not a bug in PHP support: `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists, that reason is a member of `GRAPH_INTEGRITY_GAPS`, and a graph-integrity gap demotes EVERY completed outcome in the scan. PHP was never the variable.",
"evidence": {
"nodegoat": {
"languages": {
"javascript": 44
},
"language_count": 1,
"questions_total": 299,
"questions_completed": 299,
"findings": 8,
"degradations": [
"vendor_semantics_unresolved"
]
},
"vampi": {
"languages": {
"python": 25
},
"language_count": 1,
"questions_total": 35,
"questions_completed": 35,
"findings": 4,
"degradations": []
},
"pmpro-n25": {
"languages": {
"php": 4425,
"javascript": 36
},
"language_count": 2,
"questions_total": 127,
"questions_completed": 0,
"findings": 19,
"degradations": [
"cross_partition_semantics_unresolved",
"partition_file_census_incomplete"
]
},
"pmpro-n100": {
"languages": {
"php": 4425,
"javascript": 36
},
"language_count": 2,
"questions_total": 520,
"questions_completed": 0,
"findings": 43,
"degradations": [
"cross_partition_semantics_unresolved",
"partition_file_census_incomplete"
]
}
},
"the_reading": "NodeGoat is JavaScript alone and completes 299 of 299. VAmPI is Python alone and completes 35 of 35. PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520, while establishing 19 and 43 findings. The only difference between the subjects that complete and the one that does not is the number of languages. NodeGoat's own degradation, `vendor_semantics_unresolved`, is a DECLARED exclusion rather than an integrity gap, which is exactly why it does not demote.",
"why_this_matters_for_the_release": "The declared v0.1 target is WordPress plugins, and a WordPress plugin is PHP plus its admin JavaScript. Every such repository is multi-language, so on the declared target the analyser can detect defects and can never admit one. That is a larger obstacle than the PHP runtime limit, and it is invisible from either subject that currently passes.",
"why_the_gap_exists": "A flow that crosses a language boundary is not modelled: PHP calling into JavaScript, or a script posting to an endpoint, is semantics the partitioned graph cannot follow. Recording that is right.",
"why_the_response_is_too_coarse": "The gap demotes every question, including one wholly inside a single PHP file with no cross-language element in its scope. This project has already fixed this bug class once, at the question level: task 12.3 made an empty-scope gap question-owned 'so it cannot block another question's comparison'. The same principle applies one level up. A partition boundary should demote the questions whose scope could cross it, not every question in the repository.",
"what_to_do": "Scope the partition boundary to the questions it can actually affect, and leave a question whose scope lies inside one partition completable. Do not simply reclassify the reason as a declared exclusion: that would silently claim cross-language flows are out of scope rather than unresolved.",
"limits": [
"Three subjects, one of them multi-language. A single-language PHP repository has not been measured and would isolate the cause further.",
"`partition_file_census_incomplete` is present on PMPro too and is also an integrity gap, so fixing only the cross-partition reason may not be enough on this subject."
]
}