diff --git a/.kiro/specs/hunter-learning-loop/research.md b/.kiro/specs/hunter-learning-loop/research.md index b8ec410..c09ac9b 100644 --- a/.kiro/specs/hunter-learning-loop/research.md +++ b/.kiro/specs/hunter-learning-loop/research.md @@ -62,10 +62,21 @@ `semantic/`), `mapping.analyze_entry_points`, lizard (function spans, nesting, cyclomatic complexity; pure Python), universal-ctags (spans across many languages; a binary), Semgrep OSS or Opengrep (pattern and taint findings as features; also a candidate to replace the quick-rule engine, a separate decision). -- **Status**: a measured comparison of lizard against our two engines on the pair corpus is running - (`~/ousast-results/fossil/span-eval/REPORT.md`); a licensing and capability brief for Semgrep CE and Opengrep is - being written (`~/ousast-results/fossil/semgrep-brief.md`). Both land as producers behind the same contract; - the ablation decides whether their columns stay. +- **Result, lizard (benchmarks/measurements/2026-10-13-span-engines/REPORT.md)**: on 610 pair files and 570 labelled + functions, lizard matches our parser engine line for line on C, Python and Java (end-line difference 0.00, + name-set Jaccard 0.97 to 1.00) and survives the 56 excerpts the cpp grammar rejects, but loses 6 of 44 labelled + JavaScript functions to naming (`$` identifiers, `function anonymous`, nested named functions, assigned function + expressions), gets TypeScript end lines wrong on 22% of labels (one by 81 lines), has no Perl reader, and is the + slowest of the four engines. Verdict: keep ours; lizard's per-function cyclomatic complexity and nesting are a + candidate feature column, not a span engine (its nesting correlates with `measure_nesting` at Spearman 0.6, so + it is not a drop-in there either). +- **Result, tree-sitter-language-pack**: the same tree-sitter, 503 of 503 spans identical with our parser engine, + 1.4x faster (it caches parsers), one 6 MB MIT wheel in place of six grammar wheels, and it adds Go, Ruby, PHP + and Perl grammars that ours handles by regex today. Adopt it as the grammar packaging in `semantic/extra.py`; + map `.c` to the c grammar first (20 of the 56 rejections clear); cache the parser. `tree-sitter-languages` is + ABI-broken under tree-sitter 0.26; universal-ctags has no pip build. +- **Pending**: the Semgrep CE and Opengrep brief (`~/ousast-results/fossil/semgrep-brief.md`). Producers land behind + the same contract; the ablation decides whether their columns stay. - **Research needed**: Semgrep registry rule licence terms for an open-source tool. ### Training data apart from the pilot diff --git a/benchmarks/measurements/2026-10-13-span-engines/REPORT.md b/benchmarks/measurements/2026-10-13-span-engines/REPORT.md new file mode 100644 index 0000000..6cfbad5 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-span-engines/REPORT.md @@ -0,0 +1,143 @@ +# Function-span engines: lizard and tree-sitter-language-pack against ours + +Date 2026-10-13. Script: `/home/mc/ousast-results/fossil/span-eval/span_eval.py` (raw numbers in `results.json`, +tables in `tables.md`, 12 s wall, 80 MB RSS, single process). Repository HEAD 062a7b7, nothing tracked was modified. + +## Setup + +- Corpus: every catalog-vendored pair file plus every `benchmarks/pairs/**/{vuln,fixed}.*` outside `private/`: + 610 files (cpp 352, python 102, java 66, javascript 48, typescript 36, perl 6). The six `*.in` pairs + (`openssl-cve-2022-1292`, `openssl-cve-2022-2068-hash_dir`, `openssl-cve-2026-31790-rsasve_generate`) are `c_rehash.in` Perl + (`sub link_hash_cert {`), although the catalog calls them cpp; they are scored as Perl for every engine. +- Labels: the catalog's `expected.function` per pair, applied to the vuln file and (with `fix_function` when set) + the fixed file: 570 (file, function) pairs. Names are reduced to the last segment of `A::b` / `a.b`. +- Engines: `ours_parser` = `source.ranges` (tree-sitter, regex fallback, ValueError on rejection); + `ours_line` = `declared_functions` + `function_bounds` (and `function_bounds` alone for the label); + `lizard` 1.24.1 `FileAnalyzer(get_extensions(["nd"]))`; `tslp` = tree-sitter-language-pack 1.22.1 with a 60-line + extractor of mine (function node kinds per language, declarator walk for C, parent-declarator name for JS). + All spans 1-based inclusive, earliest declaration per name. 10 s SIGALRM cap per file per engine: nothing timed out. + +## Install footprint and licences (project venv, `uv pip install`) + +| candidate | version | wheel | installed | pulls | licence | state | +|---|---|---|---|---|---|---| +| lizard | 1.24.1 | 1.2 MB | 0.4 MB | pygments, pathspec (both already present) | MIT | works | +| tree-sitter-language-pack | 1.22.1 | 6.0 MB | 6.1 MB | tree-sitter>=0.23 (0.26.0 present); 372 grammars | MIT (grammars carry their own, mostly MIT) | works | +| tree-sitter-languages | 1.10.2 | 82 MB | 85 MB | tree-sitter (no upper bound) | Apache-2.0 | `get_parser` raises `TypeError: __init__() takes exactly 1 argument` under tree-sitter 0.26 (built for <=0.21); uninstalled again | +| universal-ctags | - | - | - | - | - | no pip build exists (`universal-ctags`, `ctags`, `ctags-bin` not on PyPI; `python-ctags3` only reads tags files); `/usr/bin/ctags` is GNU Emacs ctags 29.3, not evaluated | + +## 1. Coverage of labelled functions (hit/total) + +| language | ours_parser | ours_line | lizard | tslp | +|---|---|---|---|---| +| cpp | 287/346 (83%) | 346/346 (100%) | 343/346 (99%) | 290/346 (84%) | +| python | 82/82 (100%) | 82/82 (100%) | 82/82 (100%) | 82/82 (100%) | +| java | 56/56 (100%) | 56/56 (100%) | 56/56 (100%) | 56/56 (100%) | +| javascript | 44/44 (100%) | 44/44 (100%) | 38/44 (86%) | 42/44 (95%) | +| typescript | 36/36 (100%) | 36/36 (100%) | 36/36 (100%) | 36/36 (100%) | +| perl | 6/6 (100%) | 6/6 (100%) | 4/6 (67%) | 0/6 (0%) | +| all | 511/570 (90%) | 570/570 (100%) | 559/570 (98%) | 506/570 (89%) | + +- The cpp gap of ours_parser and tslp is one cause: the tree-sitter **cpp** grammar rejects 56 files (28 pairs: + `benchmarks/pairs/vfc/curl-cve-{2020-8284,2020-8286,2021-22890,2021-22901-ossl_connect_step1,2021-22924-servercert,2022-27775,2022-27776,2022-27778,2022-27782-curl_init_userdefined,2022-27782-nss_setup_connect,2022-27782-ossl_connect_step1,2023-38545,2023-38545-do_socks5,2026-9080}/{vuln,fixed}.c` + and `openssl-cve-{2021-3449-init_sig_algs_cert,2021-3712,2021-3712-append_ia5,2021-3712-print_qualifiers,2022-1343,2022-3358,2022-3358-evp_cipher_init_internal,2023-0216,2023-0464,2023-0464-tree_init,2023-5363,2024-2511,2026-14456-expect_quic_for_value,2026-45447}/{vuln,fixed}.c`). + 20 of the 56 parse clean under the **c** grammar; ours maps `.c` to cpp. That is a grammar-selection fix in ours, not a library question. +- lizard's two cpp misses are label defects (`openssl-cve-2023-0216:STACK_OF`, `openssl-cve-2025-69421/fixed.c:OSSL_TRACE_BEGIN` + are macros, not functions); ours_line "hits" them through the brace matcher. lizard has no Perl reader (its 4/6 are + accidental brace matches at the wrong start line). tslp's Perl grammar rejects the `.in` files. +- lizard JS misses are real: `vfc-js/glance-3-0-0:Glance$serveRequest` (tokenised as `$serveRequest`), + `vfc-js/hot-formula-parser-3-0-0:performAction` (`performAction: function anonymous(` is named `anonymous`), + `vfc-js/static-eval-1-1-1:walk` (a named function nested in `module.exports = function` is reported as `(anonymous)`); + `vfc-js/underscore-1-13-0-0:template` resolves to the wrong `template` (lizard does not name `_.template = function`). + tslp misses `performAction` for the same reason (uses the `function anonymous` name over the property key; ours prefers the declarator). + +## 2. Span agreement where both engines find the label + +| pair | language | n | start= | end MAD | end<=1 | +|---|---|---|---|---|---| +| ours_parser~lizard | cpp | 286 | 100% | 0.00 | 100% | +| ours_parser~lizard | python | 82 | 100% | 0.00 | 100% | +| ours_parser~lizard | java | 56 | 0% (all -1) | 0.00 | 100% | +| ours_parser~lizard | javascript | 38 | 95% | 0.16 | 95% | +| ours_parser~lizard | typescript | 36 | 100% | 4.94 | 78% | +| ours_parser~lizard | all | 502 | 88% | 0.37 | 98% | +| ours_parser~tslp | cpp/python/java/javascript/typescript | 503 | 100% | 0.00 | 100% | +| ours_line~lizard | cpp | 343 | 100% | 0.00 | 100% | +| ours_line~lizard | python | 82 | 100% | 0.34 | 99% | +| ours_line~lizard | java | 56 | 100% | 1.00 | 100% | +| ours_line~lizard | javascript | 38 | 95% | 9.05 | 71% | +| ours_line~lizard | typescript | 36 | 100% | 2.89 | 78% | +| ours_line~lizard | all | 559 | 99% | 0.96 | 96% | +| ours_line~tslp | javascript | 42 | 100% | 8.05 | 79% | +| ours_line~tslp | typescript | 36 | 100% | 7.06 | 89% | +| ours_line~tslp | all | 506 | 89% | 1.34 | 97% | + +- Java "0% start" is a convention, not an error: tree-sitter (ours_parser, tslp) starts a method on its `@Override` + line, lizard and ours_line on the signature line; every case differs by exactly one line. +- ours_parser and tslp agree on every one of 503 spans: they run the same grammars. +- lizard's TypeScript ends are unreliable: `agent-vfc/melodix-playerprovider-089931/vuln.tsx:PlayerProvider` ends at + 46 where the arrow function ends at 127; `mercado-do-vale-news-catalogmetadataservice-9b5fcf/vuln.ts:fetchMetadata` + ends 3 lines short; `supplieradvisor-mvp-route-a69e8f/vuln.ts:GET` 3 lines short. +- ours_line is the weakest on JS/TS ends: `vfc-js/clamscan-1-2-0/vuln.js:local_scan` 674-830 (parser and lizard: 674-714), + `PlayerProvider` 13-27 (parser: 13-127), `apex-publish-static-files-2-0-0/fixed.js:publish` 16-44 (parser: 16-94). + +## 3. Name-set Jaccard per file (mean) + +| pair | cpp | python | java | javascript | typescript | perl | all | +|---|---|---|---|---|---|---|---| +| ours_parser~lizard | 0.97 (n=296) | 1.00 (n=102) | 1.00 (n=66) | 0.87 (n=47) | 0.96 (n=36) | 0.50 (n=6) | 0.97 (n=553) | +| ours_parser~tslp | 0.98 (n=296) | 1.00 (n=102) | 1.00 (n=66) | 0.94 (n=47) | 0.99 (n=36) | - | 0.99 (n=547) | +| ours_line~lizard | 0.00 (n=352) | 1.00 (n=102) | 1.00 (n=66) | 0.63 (n=47) | 0.91 (n=36) | 0.00 (n=4) | 0.38 (n=607) | +| ours_line~tslp | 0.00 (n=296) | 1.00 (n=102) | 1.00 (n=66) | 0.63 (n=47) | 0.94 (n=36) | - | 0.42 (n=547) | + +`declared_functions` has no C/C++ pattern (returns `[]`), so the line engine enumerates nothing on 58% of this corpus; +it only resolves a C function when asked for it by name. + +## 4. Runtime and failures (610 files, 7 GB host with a replay running) + +| engine | s per 1,000 files | failures (parse rejections; no crash, no timeout) | +|---|---|---| +| ours_parser | 4.2 | 56 (cpp grammar, listed above) | +| ours_line | 1.2 | 0 | +| lizard | 6.2 | 0 | +| tslp | 2.9 | 62 (the same 56 + 6 Perl) | + +ours_parser builds a `Parser` per call; tslp caches one per language, which is the 4.2 vs 2.9 difference. + +## 5. lizard nesting / cyclomatic complexity against `measure_nesting` (file level, Spearman) + +| set | n | rho(max `max_nesting_depth`) | rho(max CCN) | rho(sum CCN) | exact equal | +|---|---|---|---|---|---| +| python | 50 | 0.61 | 0.52 | 0.47 | 6/50 | +| javascript+typescript (48 JS files, padded with TS) | 50 | 0.64 | 0.60 | 0.65 | 11/50 | + +`measure_nesting` is a file-level indent/brace depth (classes and `try` count); lizard's `max_nesting_depth` is +per-function control-flow depth. Moderate rank correlation, values rarely equal: it is not a drop-in for the +`_NESTING_COEFF` term without recalibrating `complexity.signals._score`. What lizard adds that ours lacks is CCN. + +## 6. Module-level sentinel + +lizard reports no module-level unit. Code lines outside every lizard unit: cpp 11%, javascript 24%, typescript 28%, +python 40%, java 55% (fields, imports, class bodies), 18% overall; 2 files have no unit at all (the Perl `.in`). +A sentinel would be what `source.ranges` already prepends: `FunctionRange("", 1, len(lines), "module")`, with +`enclosing()` picking the innermost unit and falling back to it; lizard's nested units overlap so innermost resolution still works. + +## Verdicts + +**lizard: keep ours.** On C, Python and Java it matches the parser engine line for line (0.00 end MAD, Jaccard +0.97-1.00) and it survives the 56 excerpts the cpp grammar rejects, which is the one thing the parser engine cannot do. +But it loses 6 of 44 labelled JavaScript functions to naming (`$`, `function anonymous`, nested named functions, +assigned function expressions), its TypeScript end lines are wrong on 22% of labels (one by 81 lines), it has no Perl +reader, and it is the slowest engine here. It cannot replace the parser engine, and replacing the line engine with it +would trade the line engine's 100% label coverage for 98% while keeping a second code path for JS/TS. Its real value is +cheap per-function CCN and nesting as extra features, not spans. + +**tree-sitter-language-pack: replace the grammar packaging, not either engine.** It is the same tree-sitter: 503/503 +identical spans with ours_parser, the same 56 rejections, 1.4x faster only because it caches parsers. One 6 MB MIT +wheel replaces six `tree-sitter-` wheels and adds Go, Ruby, PHP (currently regex-only in ours) and Perl grammars. +The name-resolution logic (`declarator_name`, `ranges_from_ir`) and the regex fallback stay ours, so this is a +`grammar_for` change. Independently of the library: map `.c` to the c grammar first (20 of 56 rejections clear), and cache the parser. + +**tree-sitter-languages: not usable** (ABI of tree-sitter <=0.21; 82 MB). **universal-ctags: not evaluable** without a +system install. **Line engine: keep**; nothing tested replaces its property of resolving a named function in +unparseable text, though its JS/TS ends are the worst of the four and its C name enumeration is empty. diff --git a/benchmarks/measurements/2026-10-13-span-engines/results.json b/benchmarks/measurements/2026-10-13-span-engines/results.json new file mode 100644 index 0000000..87bd711 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-span-engines/results.json @@ -0,0 +1,1016 @@ +{ + "files": 610, + "labels": 570, + "runtime_s_per_1000": { + "ours_parser": 4.211489868838889, + "ours_line": 1.1895652253516629, + "lizard": 6.240230225041875, + "tslp": 2.9363752468718123 + }, + "failures": { + "ours_parser": [ + [ + "benchmarks/pairs/vfc/curl-cve-2020-8284/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2020-8284/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2020-8284/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2020-8284/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2020-8286/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2020-8286/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2020-8286/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2020-8286/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22890/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2021-22890/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22890/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2021-22890/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22901-ossl_connect_step1/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2021-22901-ossl_connect_step1/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22901-ossl_connect_step1/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2021-22901-ossl_connect_step1/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22924-servercert/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2021-22924-servercert/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22924-servercert/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2021-22924-servercert/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27775/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27775/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27775/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27775/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27776/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27776/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27776/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27776/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27778/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27778/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27778/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27778/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-curl_init_userdefined/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27782-curl_init_userdefined/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-curl_init_userdefined/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27782-curl_init_userdefined/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-nss_setup_connect/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27782-nss_setup_connect/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-nss_setup_connect/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27782-nss_setup_connect/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-ossl_connect_step1/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27782-ossl_connect_step1/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-ossl_connect_step1/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2022-27782-ossl_connect_step1/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2023-38545/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2023-38545/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2026-9080/fixed.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2026-9080/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2026-9080/vuln.c", + "ValueError: benchmarks/pairs/vfc/curl-cve-2026-9080/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3712/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3712/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-1343/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2022-1343/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-1343/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2022-1343/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2022-3358/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2022-3358/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0216/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-0216/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0216/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-0216/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-0464/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-0464/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-5363/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-5363/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-5363/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2023-5363/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2024-2511/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2024-2511/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2024-2511/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2024-2511/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/vuln.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-45447/fixed.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2026-45447/fixed.c: source parse failed" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-45447/vuln.c", + "ValueError: benchmarks/pairs/vfc/openssl-cve-2026-45447/vuln.c: source parse failed" + ] + ], + "ours_line": [], + "lizard": [], + "tslp": [ + [ + "benchmarks/pairs/vfc/curl-cve-2020-8284/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2020-8284/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2020-8286/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2020-8286/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22890/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22890/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22901-ossl_connect_step1/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22901-ossl_connect_step1/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22924-servercert/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2021-22924-servercert/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27775/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27775/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27776/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27776/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27778/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27778/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-curl_init_userdefined/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-curl_init_userdefined/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-nss_setup_connect/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-nss_setup_connect/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-ossl_connect_step1/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2022-27782-ossl_connect_step1/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2026-9080/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/curl-cve-2026-9080/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-1292/fixed.in", + "ValueError: no grammar for perl" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-1292/vuln.in", + "ValueError: no grammar for perl" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-1343/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-1343/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-2068-hash_dir/fixed.in", + "ValueError: no grammar for perl" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-2068-hash_dir/vuln.in", + "ValueError: no grammar for perl" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0216/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0216/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-5363/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2023-5363/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2024-2511/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2024-2511/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/vuln.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-31790-rsasve_generate/fixed.in", + "ValueError: no grammar for perl" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-31790-rsasve_generate/vuln.in", + "ValueError: no grammar for perl" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-45447/fixed.c", + "ValueError: parse error" + ], + [ + "benchmarks/pairs/vfc/openssl-cve-2026-45447/vuln.c", + "ValueError: parse error" + ] + ] + }, + "coverage": { + "cpp": { + "ours_parser": [ + 287, + 346 + ], + "ours_line": [ + 346, + 346 + ], + "lizard": [ + 343, + 346 + ], + "tslp": [ + 290, + 346 + ] + }, + "all": { + "ours_parser": [ + 511, + 570 + ], + "ours_line": [ + 570, + 570 + ], + "lizard": [ + 559, + 570 + ], + "tslp": [ + 506, + 570 + ] + }, + "perl": { + "ours_parser": [ + 6, + 6 + ], + "ours_line": [ + 6, + 6 + ], + "lizard": [ + 4, + 6 + ], + "tslp": [ + 0, + 6 + ] + }, + "python": { + "ours_parser": [ + 82, + 82 + ], + "ours_line": [ + 82, + 82 + ], + "lizard": [ + 82, + 82 + ], + "tslp": [ + 82, + 82 + ] + }, + "javascript": { + "ours_parser": [ + 44, + 44 + ], + "ours_line": [ + 44, + 44 + ], + "lizard": [ + 38, + 44 + ], + "tslp": [ + 42, + 44 + ] + }, + "typescript": { + "ours_parser": [ + 36, + 36 + ], + "ours_line": [ + 36, + 36 + ], + "lizard": [ + 36, + 36 + ], + "tslp": [ + 36, + 36 + ] + }, + "java": { + "ours_parser": [ + 56, + 56 + ], + "ours_line": [ + 56, + 56 + ], + "lizard": [ + 56, + 56 + ], + "tslp": [ + 56, + 56 + ] + } + }, + "agreement": { + "ours_parser~lizard": { + "cpp": { + "n": 286, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "all": { + "n": 502, + "start_eq": 0.8844621513944223, + "end_mad": 0.3665338645418327, + "end_within_1": 0.9800796812749004 + }, + "perl": { + "n": 4, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "python": { + "n": 82, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "javascript": { + "n": 38, + "start_eq": 0.9473684210526315, + "end_mad": 0.15789473684210525, + "end_within_1": 0.9473684210526315 + }, + "typescript": { + "n": 36, + "start_eq": 1.0, + "end_mad": 4.944444444444445, + "end_within_1": 0.7777777777777778 + }, + "java": { + "n": 56, + "start_eq": 0.0, + "end_mad": 0, + "end_within_1": 1.0 + } + }, + "ours_parser~tslp": { + "cpp": { + "n": 287, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "all": { + "n": 503, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "python": { + "n": 82, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "javascript": { + "n": 42, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "typescript": { + "n": 36, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "java": { + "n": 56, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + } + }, + "ours_line~lizard": { + "cpp": { + "n": 343, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "all": { + "n": 559, + "start_eq": 0.9892665474060823, + "end_mad": 0.9570661896243292, + "end_within_1": 0.962432915921288 + }, + "perl": { + "n": 4, + "start_eq": 0.0, + "end_mad": 0.75, + "end_within_1": 0.75 + }, + "python": { + "n": 82, + "start_eq": 1.0, + "end_mad": 0.34146341463414637, + "end_within_1": 0.9878048780487805 + }, + "javascript": { + "n": 38, + "start_eq": 0.9473684210526315, + "end_mad": 9.052631578947368, + "end_within_1": 0.7105263157894737 + }, + "typescript": { + "n": 36, + "start_eq": 1.0, + "end_mad": 2.888888888888889, + "end_within_1": 0.7777777777777778 + }, + "java": { + "n": 56, + "start_eq": 1.0, + "end_mad": 1, + "end_within_1": 1.0 + } + }, + "ours_line~tslp": { + "cpp": { + "n": 290, + "start_eq": 1.0, + "end_mad": 0, + "end_within_1": 1.0 + }, + "all": { + "n": 506, + "start_eq": 0.8893280632411067, + "end_mad": 1.3359683794466404, + "end_within_1": 0.9723320158102767 + }, + "python": { + "n": 82, + "start_eq": 1.0, + "end_mad": 0.34146341463414637, + "end_within_1": 0.9878048780487805 + }, + "javascript": { + "n": 42, + "start_eq": 1.0, + "end_mad": 8.047619047619047, + "end_within_1": 0.7857142857142857 + }, + "typescript": { + "n": 36, + "start_eq": 1.0, + "end_mad": 7.055555555555555, + "end_within_1": 0.8888888888888888 + }, + "java": { + "n": 56, + "start_eq": 0.0, + "end_mad": 1, + "end_within_1": 1.0 + } + } + }, + "jaccard": { + "ours_parser~lizard": { + "cpp": { + "n": 296, + "mean": 0.9738738738738739 + }, + "all": { + "n": 553, + "mean": 0.966793782841232 + }, + "javascript": { + "n": 47, + "mean": 0.8650214793669975 + }, + "python": { + "n": 102, + "mean": 1.0 + }, + "typescript": { + "n": 36, + "mean": 0.9642857142857143 + }, + "java": { + "n": 66, + "mean": 1.0 + }, + "perl": { + "n": 6, + "mean": 0.5 + } + }, + "ours_parser~tslp": { + "cpp": { + "n": 296, + "mean": 0.984009009009009 + }, + "all": { + "n": 547, + "mean": 0.9859327196245145 + }, + "javascript": { + "n": 47, + "mean": 0.9430690479501493 + }, + "python": { + "n": 102, + "mean": 1.0 + }, + "typescript": { + "n": 36, + "mean": 0.9920634920634921 + }, + "java": { + "n": 66, + "mean": 1.0 + } + }, + "ours_line~lizard": { + "cpp": { + "n": 352, + "mean": 0.0 + }, + "all": { + "n": 607, + "mean": 0.37971662305778986 + }, + "javascript": { + "n": 47, + "mean": 0.6334830740806962 + }, + "python": { + "n": 102, + "mean": 1.0 + }, + "typescript": { + "n": 36, + "mean": 0.9087301587301587 + }, + "java": { + "n": 66, + "mean": 1.0 + }, + "perl": { + "n": 4, + "mean": 0.0 + } + }, + "ours_line~tslp": { + "cpp": { + "n": 296, + "mean": 0.0 + }, + "all": { + "n": 547, + "mean": 0.42291456333154365 + }, + "javascript": { + "n": 47, + "mean": 0.6302123495333758 + }, + "python": { + "n": 102, + "mean": 1.0 + }, + "typescript": { + "n": 36, + "mean": 0.9365079365079365 + }, + "java": { + "n": 66, + "mean": 1.0 + } + } + }, + "nesting": { + "python": { + "n": 50, + "spearman_max_nesting_depth": 0.61018251197266, + "spearman_max_ccn": 0.5217231522987315, + "spearman_sum_ccn": 0.4691059685541511, + "exact_equal_max_nd": 6 + }, + "javascript": { + "n": 50, + "spearman_max_nesting_depth": 0.6386202078849221, + "spearman_max_ccn": 0.6024274418026722, + "spearman_sum_ccn": 0.6507917848851569, + "exact_equal_max_nd": 11 + } + }, + "module_share": { + "cpp": { + "module_code_lines": 3764, + "code_lines": 35076, + "files_without_units": 0 + }, + "all": { + "module_code_lines": 9509, + "code_lines": 51810, + "files_without_units": 2 + }, + "javascript": { + "module_code_lines": 1790, + "code_lines": 7399, + "files_without_units": 0 + }, + "python": { + "module_code_lines": 1156, + "code_lines": 2918, + "files_without_units": 0 + }, + "typescript": { + "module_code_lines": 740, + "code_lines": 2669, + "files_without_units": 0 + }, + "java": { + "module_code_lines": 1897, + "code_lines": 3452, + "files_without_units": 0 + }, + "perl": { + "module_code_lines": 162, + "code_lines": 296, + "files_without_units": 2 + } + }, + "candidate_misses": { + "tslp": [ + "openssl-cve-2021-3712:benchmarks/pairs/vfc/openssl-cve-2021-3712/vuln.c:append_ia5", + "openssl-cve-2021-3712:benchmarks/pairs/vfc/openssl-cve-2021-3712/fixed.c:append_ia5", + "openssl-cve-2022-1292:benchmarks/pairs/vfc/openssl-cve-2022-1292/vuln.in:link_hash_cert", + "openssl-cve-2022-1292:benchmarks/pairs/vfc/openssl-cve-2022-1292/fixed.in:link_hash_cert", + "curl-cve-2023-38545:benchmarks/pairs/vfc/curl-cve-2023-38545/vuln.c:do_SOCKS5", + "curl-cve-2023-38545:benchmarks/pairs/vfc/curl-cve-2023-38545/fixed.c:do_SOCKS5", + "openssl-cve-2026-14456-expect_quic_for_value:benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/vuln.c:expect_quic_for_value", + "openssl-cve-2026-14456-expect_quic_for_value:benchmarks/pairs/vfc/openssl-cve-2026-14456-expect_quic_for_value/fixed.c:expect_quic_for_value", + "openssl-cve-2026-45447:benchmarks/pairs/vfc/openssl-cve-2026-45447/vuln.c:PKCS7_verify", + "openssl-cve-2026-45447:benchmarks/pairs/vfc/openssl-cve-2026-45447/fixed.c:PKCS7_verify", + "openssl-cve-2026-31790-rsasve_generate:benchmarks/pairs/vfc/openssl-cve-2026-31790-rsasve_generate/vuln.in:rsasve_generate", + "openssl-cve-2026-31790-rsasve_generate:benchmarks/pairs/vfc/openssl-cve-2026-31790-rsasve_generate/fixed.in:rsasve_generate", + "openssl-cve-2024-2511:benchmarks/pairs/vfc/openssl-cve-2024-2511/vuln.c:ssl_update_cache", + "openssl-cve-2024-2511:benchmarks/pairs/vfc/openssl-cve-2024-2511/fixed.c:ssl_update_cache", + "openssl-cve-2023-5363:benchmarks/pairs/vfc/openssl-cve-2023-5363/vuln.c:evp_cipher_init_internal", + "openssl-cve-2023-5363:benchmarks/pairs/vfc/openssl-cve-2023-5363/fixed.c:evp_cipher_init_internal", + "openssl-cve-2023-0464:benchmarks/pairs/vfc/openssl-cve-2023-0464/vuln.c:tree_init", + "openssl-cve-2023-0464:benchmarks/pairs/vfc/openssl-cve-2023-0464/fixed.c:tree_init", + "openssl-cve-2023-0464-tree_init:benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/vuln.c:tree_init", + "openssl-cve-2023-0464-tree_init:benchmarks/pairs/vfc/openssl-cve-2023-0464-tree_init/fixed.c:tree_init", + "openssl-cve-2023-0216:benchmarks/pairs/vfc/openssl-cve-2023-0216/vuln.c:STACK_OF", + "openssl-cve-2023-0216:benchmarks/pairs/vfc/openssl-cve-2023-0216/fixed.c:STACK_OF", + "openssl-cve-2022-3358:benchmarks/pairs/vfc/openssl-cve-2022-3358/vuln.c:evp_md_init_internal", + "openssl-cve-2022-3358:benchmarks/pairs/vfc/openssl-cve-2022-3358/fixed.c:evp_md_init_internal", + "openssl-cve-2022-3358-evp_cipher_init_internal:benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/vuln.c:evp_cipher_init_internal", + "openssl-cve-2022-3358-evp_cipher_init_internal:benchmarks/pairs/vfc/openssl-cve-2022-3358-evp_cipher_init_internal/fixed.c:evp_cipher_init_internal", + "openssl-cve-2022-2068-hash_dir:benchmarks/pairs/vfc/openssl-cve-2022-2068-hash_dir/vuln.in:hash_dir", + "openssl-cve-2022-2068-hash_dir:benchmarks/pairs/vfc/openssl-cve-2022-2068-hash_dir/fixed.in:hash_dir", + "openssl-cve-2022-1343:benchmarks/pairs/vfc/openssl-cve-2022-1343/vuln.c:ocsp_verify_signer", + "openssl-cve-2022-1343:benchmarks/pairs/vfc/openssl-cve-2022-1343/fixed.c:ocsp_verify_signer", + "openssl-cve-2021-3712-append_ia5:benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/vuln.c:append_ia5", + "openssl-cve-2021-3712-append_ia5:benchmarks/pairs/vfc/openssl-cve-2021-3712-append_ia5/fixed.c:append_ia5", + "openssl-cve-2021-3712-print_qualifiers:benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/vuln.c:print_qualifiers", + "openssl-cve-2021-3712-print_qualifiers:benchmarks/pairs/vfc/openssl-cve-2021-3712-print_qualifiers/fixed.c:print_qualifiers", + "openssl-cve-2021-3449-init_sig_algs_cert:benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/vuln.c:init_sig_algs_cert", + "openssl-cve-2021-3449-init_sig_algs_cert:benchmarks/pairs/vfc/openssl-cve-2021-3449-init_sig_algs_cert/fixed.c:init_sig_algs_cert", + "curl-cve-2026-9080:benchmarks/pairs/vfc/curl-cve-2026-9080/vuln.c:mev_forget_socket", + "curl-cve-2026-9080:benchmarks/pairs/vfc/curl-cve-2026-9080/fixed.c:mev_forget_socket", + "curl-cve-2023-38545-do_socks5:benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/vuln.c:do_SOCKS5", + "curl-cve-2023-38545-do_socks5:benchmarks/pairs/vfc/curl-cve-2023-38545-do_socks5/fixed.c:do_SOCKS5" + ], + "lizard": [ + "openssl-cve-2025-69421:benchmarks/pairs/vfc/openssl-cve-2025-69421/fixed.c:OSSL_TRACE_BEGIN", + "openssl-cve-2026-31790-rsasve_generate:benchmarks/pairs/vfc/openssl-cve-2026-31790-rsasve_generate/vuln.in:rsasve_generate", + "openssl-cve-2026-31790-rsasve_generate:benchmarks/pairs/vfc/openssl-cve-2026-31790-rsasve_generate/fixed.in:rsasve_generate", + "openssl-cve-2023-0216:benchmarks/pairs/vfc/openssl-cve-2023-0216/vuln.c:STACK_OF", + "openssl-cve-2023-0216:benchmarks/pairs/vfc/openssl-cve-2023-0216/fixed.c:STACK_OF", + "glance-3-0-0:benchmarks/pairs/vfc-js/glance-3-0-0/vuln.js:Glance$serveRequest", + "glance-3-0-0:benchmarks/pairs/vfc-js/glance-3-0-0/fixed.js:Glance$serveRequest", + "hot-formula-parser-3-0-0:benchmarks/pairs/vfc-js/hot-formula-parser-3-0-0/vuln.js:performAction", + "hot-formula-parser-3-0-0:benchmarks/pairs/vfc-js/hot-formula-parser-3-0-0/fixed.js:performAction", + "static-eval-1-1-1:benchmarks/pairs/vfc-js/static-eval-1-1-1/vuln.js:walk", + "static-eval-1-1-1:benchmarks/pairs/vfc-js/static-eval-1-1-1/fixed.js:walk" + ] + }, + "lang_counts": { + "cpp": 352, + "java": 66, + "javascript": 48, + "perl": 6, + "python": 102, + "typescript": 36 + } +} \ No newline at end of file diff --git a/benchmarks/measurements/2026-10-13-span-engines/span_eval.py b/benchmarks/measurements/2026-10-13-span-engines/span_eval.py new file mode 100644 index 0000000..10d9149 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-span-engines/span_eval.py @@ -0,0 +1,469 @@ +"""Compare the project's two function-span engines against lizard and tree-sitter-language-pack. + +Run with the project venv from the repository root: + .venv/bin/python /home/mc/ousast-results/fossil/span-eval/span_eval.py +Writes results.json and tables.md next to this file. Single process, modest memory, 10 s cap per file per engine. +""" + +from __future__ import annotations + +import json +import re +import signal +import statistics +import sys +import time +import traceback +from collections import defaultdict +from pathlib import Path + +REPO = Path("/home/mc/Source/OpenUltraSAST") +OUT = Path(__file__).resolve().parent +sys.path.insert(0, str(REPO / "src")) + +import lizard # noqa: E402 +import tree_sitter_language_pack as tslp # noqa: E402 +from lizard_ext import lizardnd # noqa: E402,F401 (registers max_nesting_depth) + +from openultrasast.complexity.signals import measure_nesting # noqa: E402 +from openultrasast.pairs import load_pair_catalog # noqa: E402 +from openultrasast.source import ( # noqa: E402 + MODULE_FUNCTION, + canonical_language, + declared_functions, + function_bounds, + ranges, + spans_named, +) + +TIMEOUT_S = 10 +EXT_LANG = { + ".py": "python", ".js": "javascript", ".mjs": "javascript", ".ts": "typescript", ".tsx": "typescript", + ".java": "java", ".c": "cpp", ".cc": "cpp", ".cpp": "cpp", ".in": "perl", ".go": "go", ".rb": "ruby", ".php": "php", +} +LIZARD_EXT = {".in": ".pl"} # lizard picks its reader by extension; the *.in pairs are Perl (c_rehash.in), which lizard has no reader for +ANON = {"", "(anonymous)", "", ""} + + +class Timeout(Exception): + pass + + +def _alarm(signum, frame): # noqa: ANN001 + raise Timeout() + + +signal.signal(signal.SIGALRM, _alarm) + + +def timed(fn, *args): + """Return (result, seconds, error-string|None); error is 'timeout' after TIMEOUT_S.""" + signal.setitimer(signal.ITIMER_REAL, TIMEOUT_S) + t0 = time.perf_counter() + try: + return fn(*args), time.perf_counter() - t0, None + except Timeout: + return None, time.perf_counter() - t0, "timeout" + except Exception as exc: # noqa: BLE001 + return None, time.perf_counter() - t0, f"{type(exc).__name__}: {str(exc)[:120]}" + finally: + signal.setitimer(signal.ITIMER_REAL, 0) + + +def _keep_earliest(out: dict, name: str, span: tuple[int, int]) -> None: + """One span per name: the earliest declaration, like source.function_bounds.""" + if name not in out or span[0] < out[name][0]: + out[name] = span + + +def base_name(name: str) -> str: + """Reduce Class::method, a.b, obj->m to the last segment, as source.function_bounds does.""" + return re.split(r"::|\.|->", name)[-1].strip() + + +# ---------------------------------------------------------------- engines: each returns {name: (start1, end1)} first span + + +def ours_parser(path: str, text: str, language: str) -> dict[str, tuple[int, int]]: + out: dict[str, tuple[int, int]] = {} + for r in ranges(path, text, language): + if r.name == MODULE_FUNCTION or r.name in ANON: + continue + _keep_earliest(out, base_name(r.name), (r.start, r.end)) + return out + + +def ours_line(path: str, lines: list[str], language: str) -> dict[str, tuple[int, int]]: + out: dict[str, tuple[int, int]] = {} + for name in declared_functions(lines, language): + b = function_bounds(lines, language, name) + if b is not None: + _keep_earliest(out, base_name(name), (b[0] + 1, b[1])) + return out + + +_LIZ = lizard.FileAnalyzer(lizard.get_extensions(["nd"])) + + +def lizard_units(path: str, text: str, language: str): + suffix = Path(path).suffix + fake = "x" + LIZARD_EXT.get(suffix, suffix) + return _LIZ.analyze_source_code(fake, text) + + +def lizard_spans(info) -> dict[str, tuple[int, int]]: + out: dict[str, tuple[int, int]] = {} + for f in info.function_list: + n = base_name(f.name) + if n in ANON: + continue + _keep_earliest(out, n, (f.start_line, f.end_line)) + return out + + +TS_LANG = {"perl": None, "python": "python", "javascript": "javascript", "typescript": "typescript", "java": "java", "cpp": "cpp", + "c": "c", "go": "go", "ruby": "ruby", "php": "php"} +TS_FUNC = { + "python": {"function_definition"}, + "javascript": {"function_declaration", "method_definition", "arrow_function", "function_expression", "generator_function_declaration"}, + "typescript": {"function_declaration", "method_definition", "arrow_function", "function_expression", "generator_function_declaration"}, + "java": {"method_declaration", "constructor_declaration"}, + "cpp": {"function_definition"}, + "c": {"function_definition"}, + "go": {"function_declaration", "method_declaration"}, + "ruby": {"method", "singleton_method"}, + "php": {"function_definition", "method_declaration"}, +} +_parsers: dict[str, object] = {} + + +def _ts_parser(language: str, path: str): + key = "tsx" if language == "typescript" and path.endswith(".tsx") else TS_LANG[language] + if key not in _parsers: + _parsers[key] = tslp.get_parser(key) + return _parsers[key] + + +def _c_name(node) -> str: + d = node.child_by_field_name("declarator") + while d is not None: + if d.type in ("identifier", "field_identifier", "qualified_identifier", "destructor_name", "operator_name", "template_function"): + return d.text.decode(errors="replace") + nxt = d.child_by_field_name("declarator") + if nxt is None: + # parenthesized_declarator / pointer_declarator without the field: take the first named child + nxt = next((c for c in d.children if c.is_named), None) + d = nxt + return "" + + +def _js_name(node) -> str: + name = node.child_by_field_name("name") + if name is not None: + return name.text.decode(errors="replace") + p = node.parent + if p is not None and p.type == "variable_declarator": + n = p.child_by_field_name("name") + return n.text.decode(errors="replace") if n is not None else "" + if p is not None and p.type == "pair": + k = p.child_by_field_name("key") + return k.text.decode(errors="replace") if k is not None else "" + if p is not None and p.type == "assignment_expression": + left = p.child_by_field_name("left") + return left.text.decode(errors="replace") if left is not None else "" + return "" + + +def tslp_spans(path: str, text: str, language: str) -> dict[str, tuple[int, int]]: + if TS_LANG.get(language) is None: + raise ValueError("no grammar for " + language) + parser = _ts_parser(language, path) + tree = parser.parse(text.encode()) + if tree.root_node.has_error: + raise ValueError("parse error") + kinds = TS_FUNC[language] + out: dict[str, tuple[int, int]] = {} + stack = [tree.root_node] + while stack: + node = stack.pop() + if node.type in kinds: + if language in ("cpp", "c"): + name = _c_name(node) + elif language in ("javascript", "typescript"): + name = _js_name(node) + else: + n = node.child_by_field_name("name") + name = n.text.decode(errors="replace") if n is not None else "" + name = base_name(name) + if name and name not in ANON and re.fullmatch(r"[A-Za-z_$][\w$]*", name): + _keep_earliest(out, name, (node.start_point[0] + 1, node.end_point[0] + 1)) + stack.extend(node.children) + return out + + +# ---------------------------------------------------------------- corpus + + +def corpus() -> tuple[list[tuple[Path, str]], list[tuple[Path, str, str]]]: + cases = load_pair_catalog() + lang_of: dict[Path, str] = {} + labels: list[tuple[Path, str, str]] = [] + for c in cases: + if not c.vendored: + continue + lang = canonical_language(c.language) + for f in (c.vuln_file, c.fixed_file): + if f.suffix == ".in": + lang = "perl" # catalog says cpp; the text is c_rehash.in Perl and ours sniffs it (resolve_language) + if lang in EXT_LANG.values(): + lang_of[f.resolve()] = lang + for row in c.expected: + if row.function: + labels.append((c.vuln_file.resolve(), row.function, c.name)) + labels.append((c.fixed_file.resolve(), c.fix_function or row.function, c.name)) + files: list[tuple[Path, str]] = [] + candidates = set(lang_of) | {f.resolve() for f in (REPO / "benchmarks/pairs").rglob("*") if re.fullmatch(r"(vuln|fixed)\.[a-z]+", f.name)} + for f in sorted(candidates): + if not f.is_file() or "/private/" in f.as_posix(): + continue + lang = lang_of.get(f.resolve()) or EXT_LANG.get(f.suffix) + if lang is None or f.stat().st_size > 1_000_000: + continue + files.append((f.resolve(), lang)) + seen = set() + labels = [l for l in labels if not (l[:2] in seen or seen.add(l[:2]))] + return files, labels + + +# ---------------------------------------------------------------- spearman without scipy + + +def _ranks(xs): + order = sorted(range(len(xs)), key=lambda i: xs[i]) + r = [0.0] * len(xs) + i = 0 + while i < len(order): + j = i + while j + 1 < len(order) and xs[order[j + 1]] == xs[order[i]]: + j += 1 + for k in range(i, j + 1): + r[order[k]] = (i + j) / 2 + 1 + i = j + 1 + return r + + +def spearman(xs, ys) -> float | None: + if len(xs) < 3: + return None + rx, ry = _ranks(xs), _ranks(ys) + mx, my = statistics.mean(rx), statistics.mean(ry) + num = sum((a - mx) * (b - my) for a, b in zip(rx, ry)) + den = (sum((a - mx) ** 2 for a in rx) * sum((b - my) ** 2 for b in ry)) ** 0.5 + return None if den == 0 else num / den + + +# ---------------------------------------------------------------- main + + +def main() -> None: + files, labels = corpus() + print(f"corpus files={len(files)} labelled (file,function)={len(labels)}", file=sys.stderr) + engines = ("ours_parser", "ours_line", "lizard", "tslp") + per_file: dict[Path, dict] = {} + runtime = {e: 0.0 for e in engines} + failures = {e: [] for e in engines} + rel = lambda p: p.relative_to(REPO).as_posix() # noqa: E731 + first_check = True + for path, lang in files: + text = path.read_text(errors="replace") + lines = text.splitlines() + if first_check: + print(f"instrument check: read {rel(path)} size={len(text)} bytes lines={len(lines)}", file=sys.stderr) + first_check = False + rec = {"lang": lang, "lines": len(lines), "spans": {}, "liz": None} + r, dt, err = timed(ours_parser, rel(path), text, lang) + runtime["ours_parser"] += dt + rec["spans"]["ours_parser"] = r + if err: + failures["ours_parser"].append((rel(path), err)) + r, dt, err = timed(ours_line, rel(path), lines, lang) + runtime["ours_line"] += dt + rec["spans"]["ours_line"] = r + if err: + failures["ours_line"].append((rel(path), err)) + info, dt, err = timed(lizard_units, rel(path), text, lang) + runtime["lizard"] += dt + if err: + failures["lizard"].append((rel(path), err)) + rec["spans"]["lizard"] = None + else: + rec["spans"]["lizard"] = lizard_spans(info) + covered = set() + for f in info.function_list: + covered.update(range(f.start_line, f.end_line + 1)) + code_lines = [i + 1 for i, l in enumerate(lines) if l.strip()] + rec["liz"] = { + "max_nd": max((f.max_nesting_depth for f in info.function_list), default=0), + "max_ccn": max((f.cyclomatic_complexity for f in info.function_list), default=0), + "sum_ccn": sum(f.cyclomatic_complexity for f in info.function_list), + "n_units": len(info.function_list), + "module_code_lines": sum(1 for i in code_lines if i not in covered), + "code_lines": len(code_lines), + } + rec["ours_nesting"] = measure_nesting(text) + r, dt, err = timed(tslp_spans, rel(path), text, lang) + runtime["tslp"] += dt + rec["spans"]["tslp"] = r + if err: + failures["tslp"].append((rel(path), err)) + per_file[path] = rec + + # 1. coverage of labelled functions, 2. span agreement + cov = defaultdict(lambda: {e: [0, 0] for e in engines}) # lang -> engine -> [hit, total] + agree = defaultdict(lambda: defaultdict(list)) # (ours, cand) -> lang -> list of (start_eq, |end diff|) + misses = defaultdict(list) + for path, function, case in labels: + rec = per_file.get(path) + if rec is None: + continue + lang = rec["lang"] + name = base_name(function) + found = {} + for e in engines: + spans = rec["spans"][e] or {} + s = spans.get(name) + if e == "ours_line" and s is None and rec["spans"][e] is not None: + b = function_bounds(path.read_text(errors="replace").splitlines(), lang, function) + s = (b[0] + 1, b[1]) if b else None + if e == "ours_parser" and s is None and rec["spans"][e] is not None: + try: + sp = spans_named(ranges(rel(path), path.read_text(errors="replace"), lang), function) + s = sp[0] if sp else None + except ValueError: + s = None + found[e] = s + cov[lang][e][1] += 1 + cov["all"][e][1] += 1 + if s is not None: + cov[lang][e][0] += 1 + cov["all"][e][0] += 1 + elif e in ("lizard", "tslp"): + misses[e].append(f"{case}:{rel(path)}:{function}") + for ours in ("ours_parser", "ours_line"): + for cand in ("lizard", "tslp"): + a, b = found[ours], found[cand] + if a and b: + for key in (lang, "all"): + agree[(ours, cand)][key].append((a[0] == b[0], abs(a[1] - b[1]))) + + # 3. name Jaccard on whole files + jac = defaultdict(lambda: defaultdict(list)) + for rec in per_file.values(): + for ours in ("ours_parser", "ours_line"): + for cand in ("lizard", "tslp"): + a, b = rec["spans"][ours], rec["spans"][cand] + if a is None or b is None: + continue + sa, sb = set(a), set(b) + if not sa and not sb: + continue + for key in (rec["lang"], "all"): + jac[(ours, cand)][key].append(len(sa & sb) / len(sa | sb)) + + # 5. lizard nesting / ccn vs measure_nesting on 50 python + 50 JS files + corr = {} + for label, pick in (("python", lambda r: r["lang"] == "python"), ("javascript", lambda r: r["lang"] in ("javascript", "typescript"))): + recs = [r for r in per_file.values() if pick(r) and r["liz"] is not None][:50] + ours_v = [r["ours_nesting"] for r in recs] + corr[label] = { + "n": len(recs), + "spearman_max_nesting_depth": spearman(ours_v, [r["liz"]["max_nd"] for r in recs]), + "spearman_max_ccn": spearman(ours_v, [r["liz"]["max_ccn"] for r in recs]), + "spearman_sum_ccn": spearman(ours_v, [r["liz"]["sum_ccn"] for r in recs]), + "exact_equal_max_nd": sum(1 for r in recs if r["ours_nesting"] == r["liz"]["max_nd"]), + } + + # 6. module-level share + module = defaultdict(lambda: [0, 0, 0]) + for rec in per_file.values(): + if rec["liz"]: + for key in (rec["lang"], "all"): + module[key][0] += rec["liz"]["module_code_lines"] + module[key][1] += rec["liz"]["code_lines"] + module[key][2] += 1 if rec["liz"]["n_units"] == 0 else 0 + + n = len(files) + out = { + "files": n, + "labels": len(labels), + "runtime_s_per_1000": {e: runtime[e] / n * 1000 for e in engines}, + "failures": failures, + "coverage": {k: {e: v for e, v in d.items()} for k, d in cov.items()}, + "agreement": { + f"{o}~{c}": { + k: {"n": len(v), "start_eq": sum(x for x, _ in v) / len(v), "end_mad": statistics.mean(y for _, y in v), + "end_within_1": sum(1 for _, y in v if y <= 1) / len(v)} + for k, v in d.items() + } + for (o, c), d in agree.items() + }, + "jaccard": {f"{o}~{c}": {k: {"n": len(v), "mean": statistics.mean(v)} for k, v in d.items()} for (o, c), d in jac.items()}, + "nesting": corr, + "module_share": {k: {"module_code_lines": v[0], "code_lines": v[1], "files_without_units": v[2]} for k, v in module.items()}, + "candidate_misses": {e: v[:40] for e, v in misses.items()}, + "lang_counts": dict(sorted(defaultdict(int, {l: sum(1 for _, x in files if x == l) for l in set(x for _, x in files)}).items())), + } + (OUT / "results.json").write_text(json.dumps(out, indent=1, default=str)) + + # markdown tables + md = [] + langs = [l for l in ("cpp", "python", "java", "javascript", "typescript", "perl", "all") if l in cov] + md.append("### Coverage of labelled functions (hit/total)\n") + md.append("| language | " + " | ".join(engines) + " |") + md.append("|---|" + "---|" * len(engines)) + for l in langs: + md.append(f"| {l} | " + " | ".join(f"{cov[l][e][0]}/{cov[l][e][1]} ({cov[l][e][0] / max(cov[l][e][1], 1):.0%})" for e in engines) + " |") + md.append("\n### Span agreement on labelled functions both found (start equal %, end mean abs diff, end within 1 line %)\n") + md.append("| pair | language | n | start= | end MAD | end<=1 |") + md.append("|---|---|---|---|---|---|") + for key, d in out["agreement"].items(): + for l in langs: + if l in d: + v = d[l] + md.append(f"| {key} | {l} | {v['n']} | {v['start_eq']:.0%} | {v['end_mad']:.2f} | {v['end_within_1']:.0%} |") + md.append("\n### Name-set Jaccard per file (mean)\n") + md.append("| pair | " + " | ".join(langs) + " |") + md.append("|---|" + "---|" * len(langs)) + for key, d in out["jaccard"].items(): + md.append(f"| {key} | " + " | ".join(f"{d[l]['mean']:.2f} (n={d[l]['n']})" if l in d else "-" for l in langs) + " |") + md.append("\n### Runtime (seconds per 1,000 files, this corpus) and failures\n") + md.append("| engine | s/1000 files | failures |") + md.append("|---|---|---|") + for e in engines: + md.append(f"| {e} | {out['runtime_s_per_1000'][e]:.1f} | {len(failures[e])} |") + md.append("\n### lizard nesting/CCN vs measure_nesting (file level, Spearman)\n") + md.append("| set | n | rho(max_nesting_depth) | rho(max CCN) | rho(sum CCN) | exact == max_nesting_depth |") + md.append("|---|---|---|---|---|---|") + for l, v in corr.items(): + f = lambda x: "-" if x is None else f"{x:.2f}" # noqa: E731 + md.append(f"| {l} | {v['n']} | {f(v['spearman_max_nesting_depth'])} | {f(v['spearman_max_ccn'])} | {f(v['spearman_sum_ccn'])} | {v['exact_equal_max_nd']} |") + md.append("\n### Code lines outside any lizard unit (module-level share)\n") + md.append("| language | module code lines | code lines | share | files with zero units |") + md.append("|---|---|---|---|---|") + for l in langs: + if l in module: + v = module[l] + md.append(f"| {l} | {v[0]} | {v[1]} | {v[0] / max(v[1], 1):.0%} | {v[2]} |") + (OUT / "tables.md").write_text("\n".join(md) + "\n") + print("\n".join(md)) + print("\nfailures:", json.dumps(failures, indent=1)) + print("\ncandidate misses (first 40):", json.dumps(out["candidate_misses"], indent=1)) + print("\nlang counts:", out["lang_counts"]) + + +if __name__ == "__main__": + try: + main() + except Exception: + traceback.print_exc() + sys.exit(1)