feat(model): first repository measurement, and the four things it found

contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.

The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.

**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.

**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.

**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.

**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.

Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.

Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.

Gates byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-09 08:46:21 +02:00
co-authored by Claude Opus 5
parent 8c387306be
commit 177c7f3719
15 changed files with 776 additions and 51 deletions
+45 -1
View File
@@ -87,7 +87,7 @@
- _Requirements: 4.1_
- _Depends: 1.3_
- [ ] 2.2 The first repository measurement
- [x] 2.2 The first repository measurement
- Run the full scan against that checkout and commit the artifact: CPG build time, query time and total
scan wall-clock as three separate numbers, peak memory, findings by rung, cost, `regions_unjudged`,
and **whether the known CVE was found and at which rung**. The build/query split is separated because
@@ -118,6 +118,50 @@
- _Requirements: 4.2_
- _Depends: 2.2_
- [ ] 2.4 A region that spans the flow, not just a function
- **2.2's finding, and the one that reshapes the phase.** VAmPI-SQLI is a request parameter reaching
`get_by_username(username)`, passed to `User.get_user(username)` in another module, interpolated into a
query and executed. Source and sink are in different functions, regions are per-function, and a
repository scan does not treat parameters as untrusted -- so neither region sees both ends and the
engine is silent on the most ordinary shape a real injection bug has.
- The CPG already has the call graph; `reachableByFlows` is not function-bounded. What is bounded is the
QUESTION the driver asks. So the work is in the region model and the query scope, not in the arbiter:
a region should be able to name an entry point and admit sinks reachable from it, with the file scope
from 2.2 kept so the answer is still attributable.
- Watch the cost: an unbounded interprocedural question over a large repository is how this gets slow
again. Measure it on vampi first, then on the envelope checkout.
- Observable: VAmPI-SQLI entailed or corroborated at `models/user_model.py:get_user`, with the scan's
wall-clock and the entailed-finding count recorded against 2.2's numbers.
- _Requirements: 4.4, 8.1_
- _Depends: 2.2_
- [ ] 2.5 Module-level regions the engine can actually ask about
- A setting in `if __name__ == '__main__':` belongs to a region the mapper calls `app.py:__main__`, but
Joern names that method `<module>`. The function filter matches nothing, so module-level configuration
is invisible. Before 2.2's file scoping it was worse, not better: the literal was reported eight times
in eight files that do not contain it.
- Observable: `app.py:17`'s `host='0.0.0.0'` reported exactly once, at that location.
- _Requirements: 4.4_
- _Depends: 2.2_
- [ ] 2.6 Access control that does not entail every public endpoint
- 7 of 9 VAmPI handlers were entailed, including `register_user` and `get_all_users`, which are meant to
be unauthenticated. The dominance rule -- unguarded, with guarded siblings -- was calibrated on pairs
where a guarded twin always existed. At repository scale every public endpoint has guarded siblings.
- This is a precision question and needs a precision measurement: on a real repository, what fraction of
entailed access-control findings are endpoints that genuinely require authorisation? Do not tune it
against VAmPI alone; that is how the +16% overfitting gap in Req 8 happened.
- Observable: a recorded precision figure before and after, on more than one checkout.
- _Requirements: 8.1, 8.2_
- _Depends: 2.2_
- [ ] 2.7 A dominance witness that names its line
- Access-control sites read `api_views/users.py:?:update_password`. Taint and config witnesses carry
`(line N)`; dominance does not, so its findings have no line and a contributor cannot open them.
- Observable: every entailed finding in a repository scan has an integer line.
- _Requirements: 5.1_
- _Depends: 2.2_
## Group 3 — Ship it
- [ ] 3.1 Contributor output