feat(model): first repository measurement, and the four things it found

contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.

The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.

**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.

**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.

**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.

**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.

Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.

Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.

Gates byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

tests/test_cpg_backend.py

@@ -162,3 +162,97 @@ def test_a_batch_result_that_omits_a_request_returns_nothing_for_it() -> None:
     backend = JoernBackend(runner=lambda c, **k: _Partial())
     rows = backend.query_batch(Path("/tmp/c.bin"), "taint", {"r1": {}, "r2": {}})
     assert rows.get("r1") == [] and "r2" not in rows
+
+
+def test_a_built_cpg_offers_the_batch_path_to_its_driver(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """The capability must be reachable through the object the driver is handed, not merely present.
+
+    ``query_batch`` was implemented, tested and committed while ``build`` returned a result carrying only
+    ``run``. The driver discovers batching by looking for ``run_batch`` on that result, found nothing, and
+    fell back to one JVM per region -- silently, because every test called ``query_batch`` directly. A
+    repository scan is what made it visible: ~14s per region-family, one ``--param function=`` invocation at
+    a time.
+    """
+    from openultrasast.cpg.backend import BEGIN, END, JoernBackend
+
+    scripts: list[list[str]] = []
+
+    def runner(command, **kwargs):  # type: ignore[no-untyped-def]
+        class _Done:
+            returncode = 0
+            stderr = ""
+            stdout = f'{BEGIN}\n{{"r1": [{{"sink": "os.system(x)"}}]}}\n{END}\n'
+
+        if "--output" in command:  # the parse step; make the artifact it promises
+            Path(command[command.index("--output") + 1]).write_text("cpg")
+        else:
+            scripts.append(list(command))
+        return _Done()
+
+    monkeypatch.setenv("OPENULTRASAST_JOERN_PROBE", "on")
+    result = JoernBackend(runner=runner).build(tmp_path)
+
+    assert result is not None
+    assert result.run_batch is not None, "a backend that can batch must expose it, or the driver cannot find it"
+    rows = result.run_batch("taint", {"r1": {"sources": ("request.args",), "sinks": ("os.system",)}})
+    assert rows["r1"], "the batch path must reach the engine, not just exist"
+    assert len(scripts) == 1, "one invocation for the batch"
+    assert not any(arg.startswith("function=") for arg in scripts[0]), "a batch carries requests, not one function"
+
+
+def test_the_engine_runs_under_a_bounded_heap(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """A JVM with no -Xmx claims a quarter of physical RAM.
+
+    That is not a tuning preference. The first repository measurements were killed under memory pressure at
+    different points, so their numbers described the host rather than the engine, and on a contributor's
+    laptop an unbounded heap is the difference between a scan running in the background and a scan the
+    machine notices.
+    """
+    from openultrasast.cpg.backend import JoernBackend
+
+    commands: list[list[str]] = []
+
+    def runner(command, **kwargs):  # type: ignore[no-untyped-def]
+        class _Done:
+            returncode = 0
+            stderr = ""
+            stdout = ""
+
+        commands.append(list(command))
+        if "--output" in command:
+            Path(command[command.index("--output") + 1]).write_text("cpg")
+        return _Done()
+
+    monkeypatch.setenv("OPENULTRASAST_JOERN_PROBE", "on")
+    monkeypatch.delenv("OPENULTRASAST_CPG_HEAP_MB", raising=False)
+    result = JoernBackend(runner=runner).build(tmp_path)
+    assert result is not None
+    result.run("taint", {})
+
+    assert commands, "nothing ran"
+    for command in commands:
+        assert any(arg.startswith("-J-Xmx") for arg in command), f"unbounded heap in {command[0]}"
+
+
+def test_the_operator_can_raise_the_heap(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
+    """A bound that cannot be lifted is a bound that gets removed. A big repository may need a big heap."""
+    from openultrasast.cpg.backend import JoernBackend
+
+    commands: list[list[str]] = []
+
+    def runner(command, **kwargs):  # type: ignore[no-untyped-def]
+        class _Done:
+            returncode = 0
+            stderr = ""
+            stdout = ""
+
+        commands.append(list(command))
+        if "--output" in command:
+            Path(command[command.index("--output") + 1]).write_text("cpg")
+        return _Done()
+
+    monkeypatch.setenv("OPENULTRASAST_JOERN_PROBE", "on")
+    monkeypatch.setenv("OPENULTRASAST_CPG_HEAP_MB", "6144")
+    JoernBackend(runner=runner).build(tmp_path)
+
+    assert "-J-Xmx6144m" in commands[0]

tests/test_model_pipeline.py

@@ -79,6 +79,22 @@ def test_an_entailed_site_becomes_a_finding_with_no_model_call() -> None:
     assert client.calls == 0
 
 
+def test_an_arbitrated_finding_names_its_file_the_way_a_candidate_does() -> None:
+    """One consumer parses both shapes, so both must be ``path:line:function``.
+
+    An arbitrated finding was ``function:line``, which the report layer read as ``path:line``: every entailed
+    finding reached a contributor with a function name where its file belonged and no line at all. A whole
+    repository scan is what made it visible -- `model:access_control:get_all_books:?`, path `get_all_books`.
+    """
+    from openultrasast.model.pipeline import scan_region
+
+    findings = scan_region(_cpg(ENTAILING), _spec(), path="models/user_model.py", function="get_user", client=_Client(), model="m")
+
+    path, _, rest = findings[0].site.partition(":")
+    assert path == "models/user_model.py", "the file, not the function"
+    assert rest.endswith(":get_user")
+
+
 def test_a_sanitized_flow_the_judge_confirms_becomes_corroborated() -> None:
     from openultrasast.model.ladder import Rung
     from openultrasast.model.pipeline import scan_region
@@ -244,3 +260,16 @@ def test_the_dominance_question_describes_the_evidence_the_model_actually_has()
     assert "siblings have" not in q, "the corroborated case has no guarded siblings to appeal to"
     assert "sibling handlers are guarded" not in q
     assert "guard" in q.lower() and "public" in q.lower(), "it must still ask whether the operation needs a guard"
+
+
+def test_a_witness_line_is_read_whatever_follows_the_number() -> None:
+    """The delimiter after the line number differs by family: `(line 17,` in one, `(line 17):` in another.
+
+    Splitting on the comma produced sites like `config.py:17): permissive literal '0.0.0.0':?` -- a line
+    number no editor can open and no consumer can compare.
+    """
+    from openultrasast.model.pipeline import _site
+
+    assert _site("config.py", "", "permissive literal '0.0.0.0' (line 17): permissive") == "config.py:17:?"
+    assert _site("a.py", "run", "flow reaches os.system (line 9, via x)") == "a.py:9:run"
+    assert _site("a.py", "run", "no line at all") == "a.py:?:run"

tests/test_model_scan.py

@@ -57,8 +57,11 @@ def test_a_failed_build_yields_an_empty_result_not_an_exception() -> None:
     assert any("cpg_build_failed" in str(d) for d in result.degradations)
 
 
-def test_the_budget_is_a_total_for_the_run_and_the_remainder_is_counted(tmp_path) -> None:
-    """Req 3.2, 3.3: silently truncating is the failure this exists to prevent.
+def test_an_exhausted_budget_withholds_the_judge_and_not_the_graph(tmp_path) -> None:
+    """Req 3.2, 3.3: the budget bounds MODEL CALLS. Arbitration is free and must keep running.
+
+    The driver used to break the region loop when the budget ran out, spending a paid limit to stop unpaid
+    work. On VAmPI that left 16 of 22 regions unexamined and missed a BOLA the graph entails for nothing.
 
     Real files on disk, because the driver enumerates candidates from them. A region pointing at a path that
     does not exist yields no candidates, no model calls, and a budget that never trips -- which would make
@@ -86,11 +89,39 @@ def test_the_budget_is_a_total_for_the_run_and_the_remainder_is_counted(tmp_path
         budget=ScanBudget(max_model_calls=5, max_regions=50),
     )
     assert result.model_calls > 0, "the driver must enumerate candidates and ask the judge about them"
-    assert result.regions_unjudged > 0
-    assert result.regions_scanned + result.regions_unjudged == 50
+    assert result.regions_scanned == 50, "every region is still arbitrated; only the judge is withheld"
+    assert result.regions_unjudged == 0
+    assert result.regions_unasked > 0, "the regions the budget could not afford a judge for are counted"
     assert any("budget_exhausted" in str(d) for d in result.degradations)
 
 
+def test_a_region_the_budget_could_not_afford_is_not_reported_as_clean(tmp_path) -> None:
+    """`unasked` is the honesty counter: silent-because-decided and silent-because-unaffordable differ."""
+    from openultrasast.model.scan import ScanBudget, scan_repository
+
+    class _Client:
+        def complete(self, **kw):  # type: ignore[no-untyped-def]
+            from openultrasast.tool_hunter import ChatResponse
+
+            return ChatResponse(content='{"vulnerable": true}')
+
+    source = "import os\n\n\ndef run(cmd):\n    os.system(cmd)\n"
+    for i in range(10):
+        (tmp_path / f"f{i}.py").write_text(source)
+    regions = tuple(_region(path=f"f{i}.py") for i in range(10))
+
+    generous = scan_repository(
+        tmp_path, regions, backend=_Backend(), client=_Client(), model="m", budget=ScanBudget(max_model_calls=500, max_regions=10)
+    )
+    starved = scan_repository(
+        tmp_path, regions, backend=_Backend(), client=_Client(), model="m", budget=ScanBudget(max_model_calls=1, max_regions=10)
+    )
+
+    assert generous.regions_unasked == 0, "nothing went unasked when the budget covered the run"
+    assert starved.regions_unasked > generous.regions_unasked
+    assert starved.regions_scanned == generous.regions_scanned == 10
+
+
 def test_the_budget_is_spent_on_the_highest_ranked_regions_first() -> None:
     from openultrasast.model.scan import ScanBudget, scan_repository
 
@@ -186,3 +217,60 @@ def test_the_driver_issues_one_query_invocation_per_kind_not_one_per_region(tmp_
     assert not any(i.startswith("single:") for i in invocations), f"unbatched calls remain: {invocations[:4]}"
     # 30 regions x 2 families would have been 60 invocations; batching makes it one per query kind.
     assert len(invocations) <= 3, f"expected one invocation per query kind, got {len(invocations)}"
+
+
+class _SlowBackend:
+    """A backend whose build and whose queries each cost a measurable, distinguishable amount of time."""
+
+    def __init__(self, build_cost: float, query_cost: float) -> None:
+        self.build_cost = build_cost
+        self.query_cost = query_cost
+
+    def available(self) -> bool:
+        return True
+
+    def build(self, root):  # type: ignore[no-untyped-def]
+        import time
+
+        from openultrasast.cpg.backend import CpgResult
+
+        time.sleep(self.build_cost)
+        return CpgResult(cpg_path=Path("cpg.bin"), run=self._run)
+
+    def _run(self, query, params):  # type: ignore[no-untyped-def]
+        import time
+
+        time.sleep(self.query_cost)
+        return []
+
+
+def test_the_scan_reports_where_its_wall_clock_went() -> None:
+    """contributor-scan 2.2/2.3: build against query is a decision input, and a total cannot answer it.
+
+    Whether a warm Joern server would buy anything depends on the ratio between a CPG build and the queries
+    over it. Reporting one figure for the scan makes that question unanswerable from an artifact.
+    """
+    from openultrasast.model.scan import scan_repository
+
+    result = scan_repository(
+        Path("/repo"),
+        (_region(),),
+        backend=_SlowBackend(build_cost=0.30, query_cost=0.10),
+        client=None,
+        model="",
+    )
+
+    assert result.build_seconds >= 0.29, "the build was not timed"
+    assert result.query_seconds >= 0.09, "the queries were not timed"
+    assert result.seconds >= result.build_seconds + result.query_seconds
+    assert result.arbitrate_seconds == round(max(result.seconds - result.build_seconds - result.query_seconds, 0.0), 2)
+
+
+def test_a_failed_build_still_reports_what_the_build_cost() -> None:
+    """A build that fails after two minutes is the most important timing in the artifact, not the least."""
+    from openultrasast.model.scan import scan_repository
+
+    result = scan_repository(Path("/repo"), (_region(),), backend=_Backend(fail=True), client=None, model="")
+
+    assert result.build_seconds >= 0.0
+    assert result.query_seconds == 0.0, "nothing was queried"

tests/test_model_taint.py

@@ -395,3 +395,23 @@ def test_parameter_flows_still_decide_when_no_modelled_source_reaches_the_sink()
     ]
     answer = verdict(_cpg(rows), _spec(), function="run")
     assert answer is not None and answer.rung is Rung.ENTAILED
+
+
+def test_a_region_scopes_its_question_to_its_own_file() -> None:
+    """A region with no enclosing function must not be answered with the whole repository.
+
+    Sending `function=""` unscoped matched every hit in the tree and the driver attributed all of it to
+    whichever region asked: one permissive literal in app.py became eight identical entailed findings in
+    eight files that do not contain it.
+    """
+    from openultrasast.model.config_value import request_params as config_params
+    from openultrasast.model.specs import config_specs, taint_specs
+    from openultrasast.model.taint import request_params as taint_params
+
+    taint = taint_params(taint_specs(language="python")["injection"], function="", file="models/user_model.py")
+    config = config_params(config_specs(language="python")["config_secrets"], function="", file="config.py")
+
+    assert taint["file"] == "models/user_model.py"
+    assert config["file"] == "config.py"
+    # The pair path stays unscoped, so the committed pair measurements remain reproducible.
+    assert taint_params(taint_specs(language="python")["injection"], function="run")["file"] == ""