feat(contributor-scan): give each family its own change context (M1b 12.2)

Access control and configuration could not complete their change context on any
repository, which left 231 of 436 questions unresolved on the measured Node case
and made two of the three shipped families unable to answer.

The gap was larger than a missing query block. The evidence pass dispatches taint
only, because only taint has an evidence vector to rank by, so these families were
never asked for context at all. A new context pass asks each family with the flag
set.

Each query describes its own scope, which the owner research required and which
matters: copying taint's projection would describe a reachability walk these
queries never perform. Dominance exports the file's methods labelled operation,
guard or sibling, because its claim is that a file contradicts itself. Configuration
exports the file's methods labelled setting or sibling, because it asks what a file
declares. Both report an empty scope explicitly rather than implying one.

A family whose context query answers nothing keeps no context and stays unresolved,
so none can complete on another family's context. A control asserts that directly.

Confirmed on a real NodeGoat graph through the production backend: both families
produce a context summary and seven scoped methods where they previously produced
nothing. All seven are siblings for that file, which is the honest answer since it
declares no access-control operation and no matched setting.

1,405 tests passed, nine skipped.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-17 11:19:30 +02:00
co-authored by Claude Opus 5
parent 4ee31142ab
commit a91551e547
6 changed files with 223 additions and 11 deletions
+2 -1
View File
@@ -404,12 +404,13 @@ Preserve all completed task evidence below.
- _Boundary: Delta and Actionability Policy_ - _Boundary: Delta and Actionability Policy_
- _Requirements: 3.1, 3.2, 3.4_ - _Requirements: 3.1, 3.2, 3.4_
- _Verified 2026-09-17:_ 1,403 tests passed, nine skipped; Ruff/format/mypy passed. The symbol is now the trailing qualified callee rather than everything before the first parenthesis, so `const preTax = eval(...)` yields `eval` and a declaration naming the call covers the assigned form. Receiver spelling is preserved, so `$wpdb->prepare` and `prepare` remain different operations. A spelling that is not a qualified name, such as a dynamic `handlers[name](x)`, yields the statement text, matches no declaration and stays ineligible exactly as before. The recorded-veto report reads the same helper rather than repeating the extraction. - _Verified 2026-09-17:_ 1,403 tests passed, nine skipped; Ruff/format/mypy passed. The symbol is now the trailing qualified callee rather than everything before the first parenthesis, so `const preTax = eval(...)` yields `eval` and a declaration naming the call covers the assigned form. Receiver spelling is preserved, so `$wpdb->prepare` and `prepare` remain different operations. A spelling that is not a qualified name, such as a dynamic `handlers[name](x)`, yields the statement text, matches no declaration and stays ineligible exactly as before. The recorded-veto report reads the same helper rather than repeating the extraction.
- [ ] 12.2 Supply per-family context producers for dominance and configuration - [x] 12.2 Supply per-family context producers for dominance and configuration
- The M1b exit run left 231 of 436 head questions reporting `change_context_incomplete` from `context_projection_unavailable` and `context_scope_empty`, because those families' queries export no context. They did not block the Node injection case, so M1b correctly left them alone, but their families stay experimental until they can answer. - The M1b exit run left 231 of 436 head questions reporting `change_context_incomplete` from `context_projection_unavailable` and `context_scope_empty`, because those families' queries export no context. They did not block the Node injection case, so M1b correctly left them alone, but their families stay experimental until they can answer.
- Give dominance and configuration their own context producers describing the actual scope of the query they accompany. Do not copy taint context into them: dominance evaluates sibling operations across a file and configuration has different scope and obligations. - Give dominance and configuration their own context producers describing the actual scope of the query they accompany. Do not copy taint context into them: dominance evaluates sibling operations across a file and configuration has different scope and obligations.
- Done when a dominance and a configuration question complete their change context on a real repository, with missing extents and unsupported reasoning still explicit. - Done when a dominance and a configuration question complete their change context on a real repository, with missing extents and unsupported reasoning still explicit.
- _Boundary: contributor-scan query/context evidence_ - _Boundary: contributor-scan query/context evidence_
- _Requirements: 2.1, 2.2, 8.1, 8.2_ - _Requirements: 2.1, 2.2, 8.1, 8.2_
- _Verified 2026-09-17:_ 1,405 tests passed, nine skipped; Ruff/format/mypy passed. The gap was larger than a missing query block: the evidence pass dispatches taint only, because only taint has an evidence vector, so these families were never asked for context at all. Each query now describes its own scope, dominance the file's methods labelled operation, guard or sibling because its claim is about a file contradicting itself, and configuration the file's methods labelled setting or sibling. Neither copies taint's reachability projection, which would describe work they never do, and both report an empty scope explicitly. A new context pass asks each family with the flag set; a family whose query answers nothing keeps no context and stays unresolved, which a control asserts so no family can complete on another's context. Confirmed on a real NodeGoat graph: both produce a context summary and seven scoped methods. Evidence: `benchmarks/measurements/2026-09-17-per-family-context.json`.
- [x] 11.6 Decide what to do about engine startup, which the session cannot remove - [x] 11.6 Decide what to do about engine startup, which the session cannot remove
- **Named by the 11.5 measurement.** A transaction-owned session amortizes engine startup across the invocations inside one transaction, and a warm changed-code push does not have enough of them: startup is 9 to 13 seconds of a 30-second budget, so the session either fails to start or costs more than it saves. Group 11 as designed cannot reach the gate. - **Named by the 11.5 measurement.** A transaction-owned session amortizes engine startup across the invocations inside one transaction, and a warm changed-code push does not have enough of them: startup is 9 to 13 seconds of a 30-second budget, so the session either fails to start or costs more than it saves. Group 11 as designed cannot reach the gate.
@@ -0,0 +1,59 @@
{
"experiment": "per-family-context-v1",
"families": [
{
"query": "taint",
"seconds": 15.17,
"answered": true,
"row_kinds": {
"answer": 11,
"operation_inventory": 8,
"operation_inventory_complete": 1
},
"has_context_summary": false,
"context_methods": 0,
"relationships": {},
"boundaries": []
},
{
"query": "dominance",
"seconds": 11.87,
"answered": true,
"row_kinds": {
"context_summary": 1,
"context_method": 7
},
"has_context_summary": true,
"context_methods": 7,
"relationships": {
"sibling": 7
},
"boundaries": []
},
{
"query": "config",
"seconds": 10.96,
"answered": true,
"row_kinds": {
"context_summary": 1,
"context_method": 7
},
"has_context_summary": true,
"context_methods": 7,
"relationships": {
"sibling": 7
},
"boundaries": []
}
],
"build": {
"exit": 0,
"bytes": 883113
},
"milestone": "M1b task 12.2",
"finding": "Dominance and configuration now produce their own change context on a real graph: a context summary plus seven context methods each, scoped to the file the question is about. Taint shows no context here because its producer runs in the evidence pass, which this probe does not use; that is unchanged.",
"note_on_relationships": "All seven are 'sibling' because contributions.js declares no access-control operation and no matched configuration setting. That is the honest answer for that file, not an empty projection: the scope was described, and nothing in it is an operation of those families.",
"limits": [
"One file of one JavaScript tree; the end-to-end effect is measured by the replay, not here."
]
}
+29 -5
View File
@@ -20,7 +20,8 @@
function: String = "", function: String = "",
file: String = "", file: String = "",
requests: String = "", requests: String = "",
requestsFile: String = "" requestsFile: String = "",
contextEvidence: String = ""
) = { ) = {
importCpg(cpgFile) importCpg(cpgFile)
@@ -33,7 +34,7 @@
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
def rowsFor(settingsS: String, functionS: String, fileS: String): List[ujson.Obj] = { def rowsFor(settingsS: String, functionS: String, fileS: String, contextS: String = ""): List[ujson.Obj] = {
val names = split(settingsS) val names = split(settingsS)
val function = functionS val function = functionS
@@ -63,7 +64,30 @@
) )
} }
rows // Change-attribution context for THIS family, describing the scope this query actually used.
// Configuration asks which settings a file declares, so the scope is that file's own methods,
// and the ones holding a matched setting are what the claim is about. It is not taint's
// reachability projection and not dominance's sibling comparison; each family's context has to
// describe its own query or it is describing work nobody did.
def contextRowsFor(): List[ujson.Obj] = {
def location(m: io.shiftleft.codepropertygraph.generated.nodes.Method, kind: String): ujson.Obj =
ujson.Obj("kind" -> "context_method", "path" -> m.filename, "function" -> m.name,
"startLine" -> m.lineNumber.getOrElse(-1), "endLine" -> m.lineNumberEnd.getOrElse(-1),
"relationship" -> kind)
val scoped = {
val all = cpg.method.filterNot(_.isExternal).l
val inFile = if (fileS.isEmpty) all else all.filter(_.filename.endsWith(fileS))
if (function.isEmpty) inFile else inFile.filter(_.name == function)
}
val settingIds = calls.map(_.method.id).toSet
val located = scoped.map(m => location(m, if (settingIds.contains(m.id)) "setting" else "sibling"))
val boundaries =
if (scoped.isEmpty) List(ujson.Obj("kind" -> "context_boundary", "reason" -> "context_scope_empty:contributor-scan"))
else Nil
ujson.Obj("kind" -> "context_summary") :: (located ++ boundaries)
}
if (contextS == "true") rows ++ contextRowsFor() else rows
} }
println("---OUSAST-CPG-BEGIN---") println("---OUSAST-CPG-BEGIN---")
@@ -71,7 +95,7 @@
val parsed = ujson.read(requestsJson).obj val parsed = ujson.read(requestsJson).obj
val answers = parsed.map { case (id, req) => val answers = parsed.map { case (id, req) =>
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("") def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
id -> ujson.Arr(rowsFor(field("settings"), field("function"), field("file")): _*) id -> ujson.Arr(rowsFor(field("settings"), field("function"), field("file"), field("contextEvidence")): _*)
} }
// A census of the graph, under a key no request id can collide with (ids are numbers). A frontend can // A census of the graph, under a key no request id can collide with (ids are numbers). A frontend can
// fail every file and STILL exit 0 with a valid, empty CPG -- `joern-parse` does not even propagate the // fail every file and STILL exit 0 with a valid, empty CPG -- `joern-parse` does not even propagate the
@@ -81,7 +105,7 @@
val census = ujson.Arr(ujson.Obj("methods" -> cpg.method.size.toString, "files" -> cpg.file.size.toString, "file_names" -> ujson.Arr.from(cpg.file.name.l))) val census = ujson.Arr(ujson.Obj("methods" -> cpg.method.size.toString, "files" -> cpg.file.size.toString, "file_names" -> ujson.Arr.from(cpg.file.name.l)))
println(ujson.write(ujson.Obj.from(answers.toSeq :+ ("__census__" -> census)))) println(ujson.write(ujson.Obj.from(answers.toSeq :+ ("__census__" -> census))))
} else { } else {
println(ujson.write(ujson.Arr(rowsFor(settings, function, file): _*))) println(ujson.write(ujson.Arr(rowsFor(settings, function, file, contextEvidence): _*)))
} }
println("---OUSAST-CPG-END---") println("---OUSAST-CPG-END---")
} }
+36 -5
View File
@@ -43,7 +43,8 @@
operationRequires: String = "", operationRequires: String = "",
dischargersByKind: String = "", dischargersByKind: String = "",
requests: String = "", requests: String = "",
requestsFile: String = "" requestsFile: String = "",
contextEvidence: String = ""
) = { ) = {
importCpg(cpgFile) importCpg(cpgFile)
@@ -66,7 +67,8 @@
functionS: String, functionS: String,
fileS: String, fileS: String,
requiresS: String, requiresS: String,
byKindS: String byKindS: String,
contextS: String = ""
): List[ujson.Obj] = { ): List[ujson.Obj] = {
val opNames = split(operationsS) val opNames = split(operationsS)
@@ -138,6 +140,34 @@
if (fileS.isEmpty) all else all.filter(_.method.filename.endsWith(fileS)) if (fileS.isEmpty) all else all.filter(_.method.filename.endsWith(fileS))
} }
// Change-attribution context for THIS family, describing the scope this query actually used.
// It is deliberately not taint's projection: dominance compares sibling operations across a
// file, so the file's own methods are the scope, the methods holding a matched operation are
// what the claim is about, and the methods holding an applicable discharger are what could
// discharge it. Exporting an entry/callee reachability walk here would describe work this
// query never performs.
def contextRowsFor(): List[ujson.Obj] = {
def location(m: io.shiftleft.codepropertygraph.generated.nodes.Method, kind: String): ujson.Obj =
ujson.Obj("kind" -> "context_method", "path" -> m.filename, "function" -> m.name,
"startLine" -> m.lineNumber.getOrElse(-1), "endLine" -> m.lineNumberEnd.getOrElse(-1),
"relationship" -> kind)
val scoped =
if (fileS.isEmpty) cpg.method.filterNot(_.isExternal).l
else cpg.method.filterNot(_.isExternal).filter(_.filename.endsWith(fileS)).l
val operationIds = opCalls.map(_.method.id).toSet
val guardIds = scoped.filter(m => m.ast.isCall.code.l.exists(code => mentionsGuard(code))).map(_.id).toSet -- operationIds
val rows = scoped.map { m =>
val kind = if (operationIds.contains(m.id)) "operation" else if (guardIds.contains(m.id)) "guard" else "sibling"
location(m, kind)
}
// An empty scope is reported, never implied: a file the frontend produced no methods for
// cannot support a statement about that file contradicting itself.
val boundaries =
if (scoped.isEmpty) List(ujson.Obj("kind" -> "context_boundary", "reason" -> "context_scope_empty:contributor-scan"))
else Nil
ujson.Obj("kind" -> "context_summary") :: (rows ++ boundaries)
}
val rows = opCalls.map { op => val rows = opCalls.map { op =>
val method = op.method val method = op.method
@@ -166,7 +196,7 @@
) )
} }
rows if (contextS == "true") rows ++ contextRowsFor() else rows
} }
println("---OUSAST-CPG-BEGIN---") println("---OUSAST-CPG-BEGIN---")
@@ -181,7 +211,8 @@
field("function"), field("function"),
field("file"), field("file"),
field("operationRequires"), field("operationRequires"),
field("dischargersByKind") field("dischargersByKind"),
field("contextEvidence")
): _* ): _*
) )
} }
@@ -193,7 +224,7 @@
val census = ujson.Arr(ujson.Obj("methods" -> cpg.method.size.toString, "files" -> cpg.file.size.toString, "file_names" -> ujson.Arr.from(cpg.file.name.l))) val census = ujson.Arr(ujson.Obj("methods" -> cpg.method.size.toString, "files" -> cpg.file.size.toString, "file_names" -> ujson.Arr.from(cpg.file.name.l)))
println(ujson.write(ujson.Obj.from(answers.toSeq :+ ("__census__" -> census)))) println(ujson.write(ujson.Obj.from(answers.toSeq :+ ("__census__" -> census))))
} else { } else {
println(ujson.write(ujson.Arr(rowsFor(operations, dischargers, function, file, operationRequires, dischargersByKind): _*))) println(ujson.write(ujson.Arr(rowsFor(operations, dischargers, function, file, operationRequires, dischargersByKind, contextEvidence): _*)))
} }
println("---OUSAST-CPG-END---") println("---OUSAST-CPG-END---")
} }
+37
View File
@@ -809,9 +809,46 @@ def _evidence_pass(
for key, languages in languages_by_pair.items(): for key, languages in languages_by_pair.items():
if len(languages) > 1: if len(languages) > 1:
out.pop(key, None) out.pop(key, None)
if context_rows is not None:
_context_pass(cpg, work, context_rows, unit=unit)
return out, len(requests) - len(out) return out, len(requests) - len(out)
def _context_pass(
cpg: Any,
work: Sequence[tuple[str, ScanRegion, ArbiterSpec]],
context_rows: dict[QuestionIdentity, list[Mapping[str, object]]],
*,
unit: str,
) -> None:
"""Collect change context for the families the evidence pass does not ask.
The evidence pass dispatches taint only, because only taint has an evidence vector to rank by.
Change context is a different question and every family needs its own: without this, dominance
and configuration questions reported `context_projection_unavailable` on every repository and
could never complete, which left 231 of 436 questions unresolved on the measured Node case.
Each family's own query supplies its own scope. A failure here leaves the family's context
absent, which the caller already treats as unresolved, so it degrades exactly as before.
"""
grouped = _grouped(list(work))
by_rid = {rid: (region, spec) for rid, region, spec in work}
for kind in ("dominance", "config"):
requests = {rid: {**dict(params), "contextEvidence": "true"} for rid, params in grouped.get(kind, {}).items()}
if not requests:
continue
answered = cpg.run_batch(kind, requests)
if answered is None:
continue
answered.pop("__census__", None)
for rid, rows in answered.items():
pair = by_rid.get(rid)
if pair is None or not isinstance(rows, list):
continue
region, spec = pair
context_rows[_identity(unit, region, spec.family)] = [r for r in rows if isinstance(r, Mapping)]
def order_by_evidence(regions: Sequence[ScanRegion], evidence: Mapping[tuple[str, str, str], Evidence]) -> list[ScanRegion]: def order_by_evidence(regions: Sequence[ScanRegion], evidence: Mapping[tuple[str, str, str], Evidence]) -> list[ScanRegion]:
"""Regions by their best pair's (tier, score), then by the static order they arrived in. """Regions by their best pair's (tier, score), then by the static order they arrived in.
+60
View File
@@ -268,3 +268,63 @@ def test_a_declared_exclusion_beside_an_integrity_failure_still_demotes(tmp_path
result = scan_with_layout(tmp_path, vendor=True, symlink=True) result = scan_with_layout(tmp_path, vendor=True, symlink=True)
assert {"vendor_semantics_unresolved", "symlink_context_unresolved"} <= set(result.scope.unresolved_boundaries) assert {"vendor_semantics_unresolved", "symlink_context_unresolved"} <= set(result.scope.unresolved_boundaries)
assert result.question_outcomes[0].status == "unresolved" assert result.question_outcomes[0].status == "unresolved"
class FamilyBackend(Backend):
"""A backend that answers each family's own batch, recording what was asked."""
def __init__(self, evidence, per_kind):
super().__init__(evidence)
self.per_kind = per_kind
self.asked = []
def build(self, root, **kwargs):
graph = super().build(root, **kwargs)
outer = self
def batch(kind, requests):
outer.asked.append((kind, tuple(sorted(p.get("contextEvidence", "") for p in requests.values()))))
if any(p.get("evidenceOnly") == "true" for p in requests.values()):
return {rid: outer.evidence.get(p["file"], []) for rid, p in requests.items()}
rows = outer.per_kind.get(kind, [])
return {"__census__": [{"methods": 1, "files": 1, "file_names": ["source.php"]}], **{rid: list(rows) for rid in requests}}
graph.run_batch = batch
return graph
def test_dominance_and_config_questions_get_their_own_change_context(tmp_path):
"""M1b 12.2: the evidence pass asks taint only, so these families never had context at all."""
from dataclasses import replace as _replace
(tmp_path / "source.js").write_text("readable\n")
produced = [row("source.js", kind="operation"), {"kind": "context_summary"}]
backend = FamilyBackend(
{"source.js": vector() + [row("source.js"), {"kind": "context_summary"}]},
{"dominance": produced, "config": produced},
)
target = _replace(region("source.js", "javascript"), families=("injection", "access_control", "config_secrets"))
result = scan_repository(tmp_path, [target], backend=backend, change_context=context("source.js"), population_complete=True)
asked = {kind for kind, _ in backend.asked}
assert {"dominance", "config"} <= asked, "both families must be asked for their own context"
# Each family is asked twice: once for context, once to execute. The context ask must set the flag.
for family in ("dominance", "config"):
assert ("true",) in [flags for kind, flags in backend.asked if kind == family], family
outcomes = {o.identity.family: o for o in result.question_outcomes}
for family in ("access_control", "config_secrets"):
assert family in outcomes, family
assert outcomes[family].reason != "change_context_incomplete", family
assert not any("context_projection_unavailable" in gap for gap in result.scope.unresolved_boundaries)
def test_a_family_whose_context_query_answers_nothing_stays_unresolved(tmp_path):
"""A family that cannot describe its own scope must not complete on somebody else's context."""
from dataclasses import replace as _replace
(tmp_path / "source.js").write_text("readable\n")
backend = FamilyBackend({"source.js": vector() + [row("source.js"), {"kind": "context_summary"}]}, {})
target = _replace(region("source.js", "javascript"), families=("injection", "access_control"))
result = scan_repository(tmp_path, [target], backend=backend, change_context=context("source.js"), population_complete=True)
outcomes = {o.identity.family: o for o in result.question_outcomes}
assert "access_control" in outcomes
assert any("context_projection_unavailable" in gap for gap in result.scope.unresolved_boundaries)