Open_Ultra_SAST/benchmarks/measurements/2026-10-13-span-engines/REPORT.md

Function-span engines: lizard and tree-sitter-language-pack against ours

Date 2026-10-13. Script: /home/mc/ousast-results/fossil/span-eval/span_eval.py (raw numbers in results.json, tables in tables.md, 12 s wall, 80 MB RSS, single process). Repository HEAD 062a7b7, nothing tracked was modified.

Setup

• Corpus: every catalog-vendored pair file plus every benchmarks/pairs/**/{vuln,fixed}.* outside private/: 610 files (cpp 352, python 102, java 66, javascript 48, typescript 36, perl 6). The six *.in pairs (openssl-cve-2022-1292, openssl-cve-2022-2068-hash_dir, openssl-cve-2026-31790-rsasve_generate) are c_rehash.in Perl (sub link_hash_cert {), although the catalog calls them cpp; they are scored as Perl for every engine.
• Labels: the catalog's expected.function per pair, applied to the vuln file and (with fix_function when set) the fixed file: 570 (file, function) pairs. Names are reduced to the last segment of A::b / a.b.
• Engines: ours_parser = source.ranges (tree-sitter, regex fallback, ValueError on rejection); ours_line = declared_functions + function_bounds (and function_bounds alone for the label); lizard 1.24.1 FileAnalyzer(get_extensions(["nd"])); tslp = tree-sitter-language-pack 1.22.1 with a 60-line extractor of mine (function node kinds per language, declarator walk for C, parent-declarator name for JS). All spans 1-based inclusive, earliest declaration per name. 10 s SIGALRM cap per file per engine: nothing timed out.

Install footprint and licences (project venv, uv pip install)

candidate	version	wheel	installed	pulls	licence	state
lizard	1.24.1	1.2 MB	0.4 MB	pygments, pathspec (both already present)	MIT	works
tree-sitter-language-pack	1.22.1	6.0 MB	6.1 MB	tree-sitter>=0.23 (0.26.0 present); 372 grammars	MIT (grammars carry their own, mostly MIT)	works
tree-sitter-languages	1.10.2	82 MB	85 MB	tree-sitter (no upper bound)	Apache-2.0	get_parser raises TypeError: __init__() takes exactly 1 argument under tree-sitter 0.26 (built for <=0.21); uninstalled again
universal-ctags	-	-	-	-	-	no pip build exists (universal-ctags, ctags, ctags-bin not on PyPI; python-ctags3 only reads tags files); /usr/bin/ctags is GNU Emacs ctags 29.3, not evaluated

1. Coverage of labelled functions (hit/total)

language	ours_parser	ours_line	lizard	tslp
cpp	287/346 (83%)	346/346 (100%)	343/346 (99%)	290/346 (84%)
python	82/82 (100%)	82/82 (100%)	82/82 (100%)	82/82 (100%)
java	56/56 (100%)	56/56 (100%)	56/56 (100%)	56/56 (100%)
javascript	44/44 (100%)	44/44 (100%)	38/44 (86%)	42/44 (95%)
typescript	36/36 (100%)	36/36 (100%)	36/36 (100%)	36/36 (100%)
perl	6/6 (100%)	6/6 (100%)	4/6 (67%)	0/6 (0%)
all	511/570 (90%)	570/570 (100%)	559/570 (98%)	506/570 (89%)

• The cpp gap of ours_parser and tslp is one cause: the tree-sitter cpp grammar rejects 56 files (28 pairs: benchmarks/pairs/vfc/curl-cve-{2020-8284,2020-8286,2021-22890,2021-22901-ossl_connect_step1,2021-22924-servercert,2022-27775,2022-27776,2022-27778,2022-27782-curl_init_userdefined,2022-27782-nss_setup_connect,2022-27782-ossl_connect_step1,2023-38545,2023-38545-do_socks5,2026-9080}/{vuln,fixed}.c and openssl-cve-{2021-3449-init_sig_algs_cert,2021-3712,2021-3712-append_ia5,2021-3712-print_qualifiers,2022-1343,2022-3358,2022-3358-evp_cipher_init_internal,2023-0216,2023-0464,2023-0464-tree_init,2023-5363,2024-2511,2026-14456-expect_quic_for_value,2026-45447}/{vuln,fixed}.c). 20 of the 56 parse clean under the c grammar; ours maps .c to cpp. That is a grammar-selection fix in ours, not a library question.
• lizard's two cpp misses are label defects (openssl-cve-2023-0216:STACK_OF, openssl-cve-2025-69421/fixed.c:OSSL_TRACE_BEGIN are macros, not functions); ours_line "hits" them through the brace matcher. lizard has no Perl reader (its 4/6 are accidental brace matches at the wrong start line). tslp's Perl grammar rejects the .in files.
• lizard JS misses are real: vfc-js/glance-3-0-0:Glance$serveRequest (tokenised as $serveRequest), vfc-js/hot-formula-parser-3-0-0:performAction (performAction: function anonymous( is named anonymous), vfc-js/static-eval-1-1-1:walk (a named function nested in module.exports = function is reported as (anonymous)); vfc-js/underscore-1-13-0-0:template resolves to the wrong template (lizard does not name _.template = function). tslp misses performAction for the same reason (uses the function anonymous name over the property key; ours prefers the declarator).

2. Span agreement where both engines find the label

pair	language	n	start=	end MAD	end<=1
ours_parser~lizard	cpp	286	100%	0.00	100%
ours_parser~lizard	python	82	100%	0.00	100%
ours_parser~lizard	java	56	0% (all -1)	0.00	100%
ours_parser~lizard	javascript	38	95%	0.16	95%
ours_parser~lizard	typescript	36	100%	4.94	78%
ours_parser~lizard	all	502	88%	0.37	98%
ours_parser~tslp	cpp/python/java/javascript/typescript	503	100%	0.00	100%
ours_line~lizard	cpp	343	100%	0.00	100%
ours_line~lizard	python	82	100%	0.34	99%
ours_line~lizard	java	56	100%	1.00	100%
ours_line~lizard	javascript	38	95%	9.05	71%
ours_line~lizard	typescript	36	100%	2.89	78%
ours_line~lizard	all	559	99%	0.96	96%
ours_line~tslp	javascript	42	100%	8.05	79%
ours_line~tslp	typescript	36	100%	7.06	89%
ours_line~tslp	all	506	89%	1.34	97%

• Java "0% start" is a convention, not an error: tree-sitter (ours_parser, tslp) starts a method on its @Override line, lizard and ours_line on the signature line; every case differs by exactly one line.
• ours_parser and tslp agree on every one of 503 spans: they run the same grammars.
• lizard's TypeScript ends are unreliable: agent-vfc/melodix-playerprovider-089931/vuln.tsx:PlayerProvider ends at 46 where the arrow function ends at 127; mercado-do-vale-news-catalogmetadataservice-9b5fcf/vuln.ts:fetchMetadata ends 3 lines short; supplieradvisor-mvp-route-a69e8f/vuln.ts:GET 3 lines short.
• ours_line is the weakest on JS/TS ends: vfc-js/clamscan-1-2-0/vuln.js:local_scan 674-830 (parser and lizard: 674-714), PlayerProvider 13-27 (parser: 13-127), apex-publish-static-files-2-0-0/fixed.js:publish 16-44 (parser: 16-94).

3. Name-set Jaccard per file (mean)

pair	cpp	python	java	javascript	typescript	perl	all
ours_parser~lizard	0.97 (n=296)	1.00 (n=102)	1.00 (n=66)	0.87 (n=47)	0.96 (n=36)	0.50 (n=6)	0.97 (n=553)
ours_parser~tslp	0.98 (n=296)	1.00 (n=102)	1.00 (n=66)	0.94 (n=47)	0.99 (n=36)	-	0.99 (n=547)
ours_line~lizard	0.00 (n=352)	1.00 (n=102)	1.00 (n=66)	0.63 (n=47)	0.91 (n=36)	0.00 (n=4)	0.38 (n=607)
ours_line~tslp	0.00 (n=296)	1.00 (n=102)	1.00 (n=66)	0.63 (n=47)	0.94 (n=36)	-	0.42 (n=547)

declared_functions has no C/C++ pattern (returns []), so the line engine enumerates nothing on 58% of this corpus; it only resolves a C function when asked for it by name.

4. Runtime and failures (610 files, 7 GB host with a replay running)

engine	s per 1,000 files	failures (parse rejections; no crash, no timeout)
ours_parser	4.2	56 (cpp grammar, listed above)
ours_line	1.2	0
lizard	6.2	0
tslp	2.9	62 (the same 56 + 6 Perl)

ours_parser builds a Parser per call; tslp caches one per language, which is the 4.2 vs 2.9 difference.

5. lizard nesting / cyclomatic complexity against measure_nesting (file level, Spearman)

set	n	rho(max max_nesting_depth)	rho(max CCN)	rho(sum CCN)	exact equal
python	50	0.61	0.52	0.47	6/50
javascript+typescript (48 JS files, padded with TS)	50	0.64	0.60	0.65	11/50

measure_nesting is a file-level indent/brace depth (classes and try count); lizard's max_nesting_depth is per-function control-flow depth. Moderate rank correlation, values rarely equal: it is not a drop-in for the _NESTING_COEFF term without recalibrating complexity.signals._score. What lizard adds that ours lacks is CCN.

6. Module-level sentinel

lizard reports no module-level unit. Code lines outside every lizard unit: cpp 11%, javascript 24%, typescript 28%, python 40%, java 55% (fields, imports, class bodies), 18% overall; 2 files have no unit at all (the Perl .in). A sentinel would be what source.ranges already prepends: FunctionRange("<module>", 1, len(lines), "module"), with enclosing() picking the innermost unit and falling back to it; lizard's nested units overlap so innermost resolution still works.

Verdicts

lizard: keep ours. On C, Python and Java it matches the parser engine line for line (0.00 end MAD, Jaccard 0.97-1.00) and it survives the 56 excerpts the cpp grammar rejects, which is the one thing the parser engine cannot do. But it loses 6 of 44 labelled JavaScript functions to naming ($, function anonymous, nested named functions, assigned function expressions), its TypeScript end lines are wrong on 22% of labels (one by 81 lines), it has no Perl reader, and it is the slowest engine here. It cannot replace the parser engine, and replacing the line engine with it would trade the line engine's 100% label coverage for 98% while keeping a second code path for JS/TS. Its real value is cheap per-function CCN and nesting as extra features, not spans.

tree-sitter-language-pack: replace the grammar packaging, not either engine. It is the same tree-sitter: 503/503 identical spans with ours_parser, the same 56 rejections, 1.4x faster only because it caches parsers. One 6 MB MIT wheel replaces six tree-sitter-<lang> wheels and adds Go, Ruby, PHP (currently regex-only in ours) and Perl grammars. The name-resolution logic (declarator_name, ranges_from_ir) and the regex fallback stay ours, so this is a grammar_for change. Independently of the library: map .c to the c grammar first (20 of 56 rejections clear), and cache the parser.

tree-sitter-languages: not usable (ABI of tree-sitter <=0.21; 82 MB). universal-ctags: not evaluable without a system install. Line engine: keep; nothing tested replaces its property of resolving a named function in unparseable text, though its JS/TS ends are the worst of the four and its C name enumeration is empty.