diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/NOTES.md b/benchmarks/measurements/2026-10-13-access-control-screen/NOTES.md new file mode 100644 index 0000000..d59f460 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/NOTES.md @@ -0,0 +1,97 @@ +# Access-control feature table and separation screen, no model calls (2026-10-13/14) + +## What was built + +`table.py` checks out the head of every change of the frozen access-control training pool (digest 53f9f364: +5 fix pushes with 48 verified locations, 100 ordinary pushes, 5 repositories), enumerates the candidate rows +(functions the push touched, plus the verified fix sites of a positive) and runs the four producers over them. +Rows are labelled 1 at a verified fix site, 0 on an ordinary push. The screen scores every engineered feature alone +against the label (ROC-AUC) and names the strongest feature per model annotation. Repositories are the clusters, +and there are five, so the screen is a direction, not a result. + +The final table: 952 rows, 74 positive rows (62 functions; a function with two verified spans counts twice), +878 negative rows, digest 02774edc (152a72ce with folds). Producer coverage: diff 952, source 933, semantic 262 +(the semantic producer handles TypeScript and JavaScript only; Java has no obligations support and PHP is +unsupported), taxonomy 952 (a constant: it renders the family list, it is not a row feature). + +## It took three builds, and the first two printed plausible numbers + +1. **Build 1** (952 rows, source ok 783): the screen printed AUCs. 150 rows, 16%, every positive of trigger.dev + among them, had read an empty file although git holds the blobs. The gate `source == 0` did not fire because + most rows were fine. +2. **Build 2**, after adding a checkout proof (`git status` clean after the sparse checkout is undone) and an + unread-row gate (every unread row checked against `git cat-file -s`): the gate refused the table, the same + 150 rows. The checkout proof never fired, so the checkout was not the cause. +3. The cause was a producer bug: the entry-point scan read every product file of the tree, one empty file raised + `empty_source`, the exception escaped the cached property and failed the source producer on every row of + the change, re-reading the whole tree per row (fcb43fd). **Build 3**: 0 unread rows, source ok 933; the 19 + remaining failures are files the push deleted (FileNotFoundError, 16) and three unparseable tsx files. + +A producer that fails on every row of whole changes while the same files read fine by hand is a producer bug, +not disk or concurrency. The memory note carries it. + +## The diff producer was blind to the shape of this class + +A positive push here is the fix replayed backwards (base = fix commit, head = its parent). A fix that adds a +permission check becomes a push that only removes lines, and the diff producer kept only hunks with added lines +at head. In build 1, 38 of 74 positive rows had no hunk in their file at all and 69 of 74 were "not on a changed +line", while 60% of ordinary rows were: the hunks annotation separated the labels by construction and showed +the model nothing at the fix site. The producer (version 2, 537b61e) now records the head lines after which the +push removed code; distance and on_changed_line count them; the model's hunks input renders them as +`removed_after_line`; `diff.deletion_count` is an engineered feature. In build 3, 66 of 74 positive rows have a +deletion anchor in their file and 12 are on a changed line. + +## Screen (build 3, `separation.csv`, `screen.csv`) + +| Annotation | Strongest feature | AUC | rows | +|---|---|---|---| +| hunks | diff.deletion_count | 0.89 | 952 | +| obligations | semantic.obligations_undischarged | 0.76 | 262 | +| callers | source.caller_count | 0.72 | 933 | +| entry_points | source.entry_point_count | 0.71 | 933 | +| roles | semantic.sinks | 0.17 | 46 | +| function_text | source.span_lines | 0.36 | 933 | +| change_summary | source.is_module | 0.36 | 933 | +| def_use | semantic.binds | 0.36 | 262 | + +Read with care: + +- **deletion_count (0.89) and the low hunk_lines (0.15) are the construction, not the class.** Reversed fixes are + deletion-heavy pushes with few added lines; ordinary pushes are not. Any feature that says "this push removed + code and added little" separates these labels. `separation_deletion_rows.csv` repeats the screen on the 235 + rows whose file lost lines (66 positive, 169 negative, all 5 repositories): there deletion_count drops to 0.71, + hunk_lines and hunk_count stay at 0.12 and 0.14, and **entry_point_count rises to 0.83, caller_count stays at + 0.73**. Those two are the class: the verified access-control sites are web entry points with callers. +- **obligations_undischarged (0.76) rests on 8 positive rows from one repository** (trigger.dev; the semantic + producer covers no Java or PHP). Not evidence yet. +- **roles** scored 46 rows. The semantic annotation is absent for 80% of this pool. +- `leak_audit.csv` is the same screen over the folded table (`ablation.leak_audit`), kept for the ablation record. + +## Folds and the first ablation manifest (registered, not run) + +Folds are repository-grouped (`ablation.assign_folds`, k=5, seed 0), one repository per fold: 0 p4-plugin (6 +positive rows), 1 dependency-track (4), 2 job-import (2), 3 shopper (54), 4 trigger.dev (8). +`ablation.json` registers the first manifest in `~/ousast-results/hunt/ac-ablation-1/` for a decision: arms +full, no-hunks, no-entry_points, no-callers, no-semantic; compile folds 0-2 (the Jenkins plugins), holdout 3-4 +(shopper and trigger.dev, 62 of the 74 positive rows); metric search_outcome at tolerance 10; one repeat; 8 +dollars and 1,500 calls per arm. The baseline cost 6.13 dollars for one arm over this pool, so this manifest is +about 35 dollars. With 5 positive pushes `analyse` will report `dry_run`; the location-level numbers are what it +can say. + +## Open questions for the maintainer + +- Whether the hunks annotation should be ablated at all while positives are reversed fixes, or whether the + ordinary draw should be stratified by "push removed lines" so the construction signal cancels. +- Whether to extend the semantic producer to Java (obligations) and PHP before the next class round; 80% of + this pool gets no roles, def_use or obligations today. +- The 21 access-control sources dropped by the disk guard and batch 2 (27 changes, 69 locations, awaiting the + spot-check) would take positives past 40. + +## Files + +- `table.py`: the build (worktree per change, full checkout, producers, labels, gates, screen). +- `summary.json`, `separation.csv`, `screen.csv`, `separation_deletion_rows.csv`, `leak_audit.csv`, `log.json` + (per-change seconds and rows; the slowest ordinary change of shopper took 2,407 s in the repository-wide + caller scan), `ablation.json`. +- Table: `~/ousast-results/valid/train/work/ac-screen/table.jsonl` (94 MB) and `table_folds.jsonl`; builds 1 and + 2 kept beside it as `ac-screen-v1` and `ac-screen-v2-refused`. diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/ablation.json b/benchmarks/measurements/2026-10-13-access-control-screen/ablation.json new file mode 100644 index 0000000..fa9297d --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/ablation.json @@ -0,0 +1 @@ +{"manifest":{"arms":{"full":[],"no-callers":["callers"],"no-entry_points":["entry_points"],"no-hunks":["hunks"],"no-semantic":["semantic"]},"budget":{"calls":1500,"usd":8.0},"compile_folds":[0,1,2],"holdout_folds":[3,4],"metric":"hunt.metric.search_outcome(tolerance=10)","program_id":"e79db45fd4a35acf001fedbd7cbb291dc9f03cdb611aafe9f1655884798d206f","rule":{"needs":{},"resamples":2000,"version":"keep-drop-v1"},"sample":{"changes":["jenkinsci-dependency-track-plugin-673cc48b4f75ebf5","jenkinsci-dependency-track-plugin-ordinary-0d167208df6d423f5170101165abbc9deb005841","jenkinsci-dependency-track-plugin-ordinary-123a5dc0150fc0e45e6e27cb36c55108dd4b2c26","jenkinsci-dependency-track-plugin-ordinary-16f7f643651266f2e0d10901152c81261ee44225","jenkinsci-dependency-track-plugin-ordinary-1efa1410b701872981b903d7857d2406913d5b97","jenkinsci-dependency-track-plugin-ordinary-2c7d0738ad3e55626021a48d7e23d4aa18163ac0","jenkinsci-dependency-track-plugin-ordinary-3b1a2ea9082a3a0e7a8cfaa9a448ae9ad23e528b","jenkinsci-dependency-track-plugin-ordinary-432f1df244aaa3a6ca02c1412a597d2dc97db5aa","jenkinsci-dependency-track-plugin-ordinary-50848c0edeeb99d3803bed63324a48754228a159","jenkinsci-dependency-track-plugin-ordinary-670f5988f7939f18343c3547d6d363e1600cc70b","jenkinsci-dependency-track-plugin-ordinary-694379976b192eaaf0275e5cbfe95e8ca898d06b","jenkinsci-dependency-track-plugin-ordinary-6d66c56feb5dc23ecc5f8f73562e06ec0c8e65ff","jenkinsci-dependency-track-plugin-ordinary-9203f702005c090bff7898ce91cafac9a26a8b7d","jenkinsci-dependency-track-plugin-ordinary-9f5ddda50d037d1185953377896c80def8d41e57","jenkinsci-dependency-track-plugin-ordinary-bbc913e3c9878ae3a56712f28cdeb6464bf5daee","jenkinsci-dependency-track-plugin-ordinary-bbe8342f2d8f4cdf4566f71cc7ed87b695f3a5c4","jenkinsci-dependency-track-plugin-ordinary-bcfa2ae89b8627657c1d2e9ddd71d5b285f49f25","jenkinsci-dependency-track-plugin-ordinary-def475c90bc85f709dc22841ffcb265c48afc5fd","jenkinsci-dependency-track-plugin-ordinary-e74042b706b640355643be4c9ab8aa0f4dddb5c4","jenkinsci-dependency-track-plugin-ordinary-e7e4409570ee0d2eb31ba7ade5ecedb5b817f436","jenkinsci-dependency-track-plugin-ordinary-fd311a6cef414e39b08f528608b9f1753b0b9a77","jenkinsci-job-import-plugin-07e0e7023a46cf4d","jenkinsci-job-import-plugin-ordinary-044a2e819b271dade0106de10291b2fed45836df","jenkinsci-job-import-plugin-ordinary-1bcce383bc2977e8ed5170eea2fdab408edbe682","jenkinsci-job-import-plugin-ordinary-23feb819c2e2778f60e8a7a8096b92d4086ff3ea","jenkinsci-job-import-plugin-ordinary-445573104f2dda90a133308d574c2fed0828d081","jenkinsci-job-import-plugin-ordinary-4e2cb93e8de3b3ffacddbf72bd157cae1ef67f46","jenkinsci-job-import-plugin-ordinary-4f97016985ec2a5080e0868261f60a644604f483","jenkinsci-job-import-plugin-ordinary-5032d8339ef734759812af9ee37b01823c0de3f4","jenkinsci-job-import-plugin-ordinary-66b17d16ecd2f108912351ddb2e15a637520b0a8","jenkinsci-job-import-plugin-ordinary-6be5a096043a669dfa535e22302c8d98d7ce5233","jenkinsci-job-import-plugin-ordinary-7e5f489c88805d8e25ade0d1c81d4fc7149abc7e","jenkinsci-job-import-plugin-ordinary-b5334bc4a6be08abd0e44e443a340d1b6f135c72","jenkinsci-job-import-plugin-ordinary-bb511a1d2e44a7c206d09e43daec7af1d9a78b2e","jenkinsci-job-import-plugin-ordinary-dc0abc56f335aa048192b1ff173da6a036d54322","jenkinsci-job-import-plugin-ordinary-e29917d2c7eb818e4ffda664a932b942e2aee2b0","jenkinsci-job-import-plugin-ordinary-e52d913aad5f83cb6c35c6494f15d77c7bce72db","jenkinsci-job-import-plugin-ordinary-ec1711f1bd0ab146225bca3057a17ae7a4154828","jenkinsci-job-import-plugin-ordinary-ec2103f6375fadb68375fd782e4de6d3daf1fccb","jenkinsci-job-import-plugin-ordinary-f10bc2f0443d6ae6d49217a4c0e6271451fd0ce5","jenkinsci-job-import-plugin-ordinary-f10f5a83cc4070533bdbe2be4e476e2220165b9d","jenkinsci-job-import-plugin-ordinary-ffdeb8e959d01e145dd1c970595f929cb562f360","jenkinsci-p4-plugin-5022afedadfe8449","jenkinsci-p4-plugin-ordinary-0cfd50a9fdc86cfe3e9a770038d273ea382f59bb","jenkinsci-p4-plugin-ordinary-171dde20917615e63a6120251d7cb367f9d52992","jenkinsci-p4-plugin-ordinary-18e4708722279a3a07eafb35f1419dfed6c7dbd1","jenkinsci-p4-plugin-ordinary-1a38001c4dc8e2424479d44f95f6e25e25df4959","jenkinsci-p4-plugin-ordinary-1d19356cb9f2d8e9b72a129e5719cca45d89821f","jenkinsci-p4-plugin-ordinary-2ec2dccaa9e6ff318ccada47f2af691170aaa7be","jenkinsci-p4-plugin-ordinary-30fb98f8e26adaa33762b526cdd443aa6122ac5b","jenkinsci-p4-plugin-ordinary-46fb68bfea093c51f841bc6bdec9676bc4d468c7","jenkinsci-p4-plugin-ordinary-4cb6dab4456e1afb3de1230cfacdce3e162fe8ea","jenkinsci-p4-plugin-ordinary-65c18b225338e93837ef6882ad79cd9773e6e2c4","jenkinsci-p4-plugin-ordinary-65fbed70350a4435ec091390d92722bbf7cc1c68","jenkinsci-p4-plugin-ordinary-7b512b08c584a76d57f4401634f8e9736a365223","jenkinsci-p4-plugin-ordinary-8a67359f1a69264da35c9e837273a649b3345b9d","jenkinsci-p4-plugin-ordinary-a33e38e16ececd6e093eb213a9b8a3ab8352f937","jenkinsci-p4-plugin-ordinary-aa910d56936a308600bdad568e2ab23ada636b7d","jenkinsci-p4-plugin-ordinary-b2f7b1ba3eea1a81370dec019ebff29931875182","jenkinsci-p4-plugin-ordinary-b42c75d6823b9e67bd968679494d1090f5d77941","jenkinsci-p4-plugin-ordinary-dca0f55c69c487611689e72230e211ef4b760d9e","jenkinsci-p4-plugin-ordinary-f4c24f7964bd1de847ace7b43546be81e51d2ccc","jenkinsci-p4-plugin-ordinary-f74fb378f0e7ce258aae449dc70f35820a1b11ed","shopperlabs-shopper-02254beed509364c","shopperlabs-shopper-ordinary-0db4faf747f7bc4d9eadacbfa0dcbd2620251ca5","shopperlabs-shopper-ordinary-13037bfd039904801dd8103dec3d0e720c547b94","shopperlabs-shopper-ordinary-17778cdd2c24a50d2fb29dae4a3aab5390b1b996","shopperlabs-shopper-ordinary-1aa9a4c2f471f00914a230b8168b3926f39837a7","shopperlabs-shopper-ordinary-31f33b1631254639836ce06ad6b078e6cfb0b13a","shopperlabs-shopper-ordinary-44809499ef406c9e199965fff6d124434857a36b","shopperlabs-shopper-ordinary-4628430c0a70506e15fba209557606f860566598","shopperlabs-shopper-ordinary-5394b647023599eb9d9a40f7b9ccc434cd8136a3","shopperlabs-shopper-ordinary-5a079da974a3d7f6e7ca0de3c6ca6b0849e304a2","shopperlabs-shopper-ordinary-7e56ccd817b0dc835919fac74db792e15ee62c1d","shopperlabs-shopper-ordinary-7eaab898afd90140f2162149585faf2642e18987","shopperlabs-shopper-ordinary-7f7e6bdb67509a50fae95f13a829494a8a3ab7b4","shopperlabs-shopper-ordinary-886e671270b21aed8ab851b5b82a53fc6e87fbc6","shopperlabs-shopper-ordinary-9313ed20032199988f571d7e655255819f7f6f04","shopperlabs-shopper-ordinary-a0c5bd3a05a0fef58b9043273d6062124a3ecf8e","shopperlabs-shopper-ordinary-a5a2760dc0c63872b55afec0e3d452df7989393b","shopperlabs-shopper-ordinary-b178fe37da1c31f9f0057f35fdf4145936a25138","shopperlabs-shopper-ordinary-ceaea6a078025d4a7634d2646d27dde3f13b6f58","shopperlabs-shopper-ordinary-fada4250aaf3350c1db92698259c54f9ae6b4bb0","shopperlabs-shopper-ordinary-fcc3888151b9c43843c01f631870dfe7e8fd6329","triggerdotdev-trigger.dev-0eb72665259b13c2","triggerdotdev-trigger.dev-ordinary-0e63f8317e7c8b472d8c9c7bb36ad3986d0238c7","triggerdotdev-trigger.dev-ordinary-22a7e33df74729c0a2dbd4b12d30c60378ec677f","triggerdotdev-trigger.dev-ordinary-2d569f64f8027bd2dde80ce14042542200f8fc7d","triggerdotdev-trigger.dev-ordinary-35298ac357d4717203cc181986d29109ae90e6cd","triggerdotdev-trigger.dev-ordinary-409388365eaaa9bb61eb4a9060e1fb46c8acc6a2","triggerdotdev-trigger.dev-ordinary-417ab876e3177ced4a4aa46c3de664b10950c6fe","triggerdotdev-trigger.dev-ordinary-478adb5d74d36f860f6f22d28ee18a299d692f45","triggerdotdev-trigger.dev-ordinary-4c42f6cc0b183a1435ec39d831892ca58b415008","triggerdotdev-trigger.dev-ordinary-55d85d0b2360cdeaecbf94f9fea8404cc5853acb","triggerdotdev-trigger.dev-ordinary-6be6c1816c2dd2dc09fc361f40be6af2a07c2de2","triggerdotdev-trigger.dev-ordinary-771937adf5d6b72943a006ec86b5bfe1057804e7","triggerdotdev-trigger.dev-ordinary-7ab437c8ad8ef1db66132157056cb59d94e763a8","triggerdotdev-trigger.dev-ordinary-87448ccaf23b7f99f505f953f2ae6fd6c3f3975c","triggerdotdev-trigger.dev-ordinary-9f29eb209042c4682f3fab90b4036dba300ea892","triggerdotdev-trigger.dev-ordinary-bfa902bd1822812e8f0aaf0d6831414dc70c13aa","triggerdotdev-trigger.dev-ordinary-cd518683c8c7830176bb1f9efdcbfe4b03d79457","triggerdotdev-trigger.dev-ordinary-cf4aa7e918eb6c3634b7dc8827b21be246519591","triggerdotdev-trigger.dev-ordinary-efcb89ac26984aac8e98f9c6ccb9a902c0483881","triggerdotdev-trigger.dev-ordinary-f78261a606362cf5480679e7286585367d744d08","triggerdotdev-trigger.dev-ordinary-fdec5a09a72fae962f3db9b3dcf8af08d34d06ad"],"repeats":1},"seed":0,"table_digest":"152a72ce845dc220faf6a8ece1018cd912080f8058efec54601c9ead9f0cb925"},"manifest_id":"68c30d2f0a4b0f2032caae16619b45f3a42943b955cc693948bac9d27db6605d"} diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/leak_audit.csv b/benchmarks/measurements/2026-10-13-access-control-screen/leak_audit.csv new file mode 100644 index 0000000..87ed5bb --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/leak_audit.csv @@ -0,0 +1,18 @@ +column,auc,n,note,annotation +diff.distance,0.7519618922057947,935,,hunks +diff.on_changed_line,0.26616080773256173,952,,hunks +diff.hunk_count,0.26391368589546266,952,,hunks +diff.hunk_lines,0.1529581973773318,952,,hunks +diff.deletion_count,0.8854275687988672,952,,hunks +source.span_lines,0.36404681748104334,933,,function_text +source.is_module,0.3577462794575717,933,,change_summary +source.caller_count,0.7206525501054023,933,,callers +source.caller_files,0.7166567032690432,933,,callers +source.entry_point_count,0.7070918415505144,933,,entry_points +semantic.sources,0.6976744186046512,46,,roles +semantic.sinks,0.17441860465116277,46,,roles +semantic.sanitizers,0.4534883720930233,46,,roles +semantic.binds,0.35999015748031493,262,,def_use +semantic.calls,0.5642224409448819,262,,def_use +semantic.obligations,0.7541830708661417,262,,obligations +semantic.obligations_undischarged,0.7620570866141732,262,,obligations diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/log.json b/benchmarks/measurements/2026-10-13-access-control-screen/log.json new file mode 100644 index 0000000..0b2d40d --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/log.json @@ -0,0 +1,737 @@ +[ + { + "id": "jenkinsci-dependency-track-plugin-673cc48b4f75ebf5", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "vulnerability", + "seconds": 0.79, + "rows": 4 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-0d167208df6d423f5170101165abbc9deb005841", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 3.55, + "rows": 9 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-123a5dc0150fc0e45e6e27cb36c55108dd4b2c26", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.29, + "rows": 1 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-16f7f643651266f2e0d10901152c81261ee44225", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.99, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-1efa1410b701872981b903d7857d2406913d5b97", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.64, + "rows": 4 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-2c7d0738ad3e55626021a48d7e23d4aa18163ac0", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.35, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-3b1a2ea9082a3a0e7a8cfaa9a448ae9ad23e528b", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.56, + "rows": 4 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-432f1df244aaa3a6ca02c1412a597d2dc97db5aa", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.49, + "rows": 3 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-50848c0edeeb99d3803bed63324a48754228a159", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.39, + "rows": 3 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-670f5988f7939f18343c3547d6d363e1600cc70b", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.79, + "rows": 6 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-694379976b192eaaf0275e5cbfe95e8ca898d06b", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.25, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-6d66c56feb5dc23ecc5f8f73562e06ec0c8e65ff", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.44, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-9203f702005c090bff7898ce91cafac9a26a8b7d", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.7, + "rows": 3 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-9f5ddda50d037d1185953377896c80def8d41e57", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.53, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-bbc913e3c9878ae3a56712f28cdeb6464bf5daee", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.9, + "rows": 3 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-bbe8342f2d8f4cdf4566f71cc7ed87b695f3a5c4", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 2.07, + "rows": 7 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-bcfa2ae89b8627657c1d2e9ddd71d5b285f49f25", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 3.92, + "rows": 16 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-def475c90bc85f709dc22841ffcb265c48afc5fd", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.42, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-e74042b706b640355643be4c9ab8aa0f4dddb5c4", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.89, + "rows": 4 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-e7e4409570ee0d2eb31ba7ade5ecedb5b817f436", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 0.86, + "rows": 2 + }, + { + "id": "jenkinsci-dependency-track-plugin-ordinary-fd311a6cef414e39b08f528608b9f1753b0b9a77", + "repo": "jenkinsci/dependency-track-plugin", + "kind": "ordinary", + "seconds": 1.92, + "rows": 11 + }, + { + "id": "jenkinsci-job-import-plugin-07e0e7023a46cf4d", + "repo": "jenkinsci/job-import-plugin", + "kind": "vulnerability", + "seconds": 0.3, + "rows": 2 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-044a2e819b271dade0106de10291b2fed45836df", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.54, + "rows": 6 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-1bcce383bc2977e8ed5170eea2fdab408edbe682", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.44, + "rows": 7 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-23feb819c2e2778f60e8a7a8096b92d4086ff3ea", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.12, + "rows": 1 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-445573104f2dda90a133308d574c2fed0828d081", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.97, + "rows": 11 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-4e2cb93e8de3b3ffacddbf72bd157cae1ef67f46", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.41, + "rows": 7 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-4f97016985ec2a5080e0868261f60a644604f483", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 3.0, + "rows": 58 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-5032d8339ef734759812af9ee37b01823c0de3f4", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.12, + "rows": 1 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-66b17d16ecd2f108912351ddb2e15a637520b0a8", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.41, + "rows": 8 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-6be5a096043a669dfa535e22302c8d98d7ce5233", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.39, + "rows": 5 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-7e5f489c88805d8e25ade0d1c81d4fc7149abc7e", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.26, + "rows": 6 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-b5334bc4a6be08abd0e44e443a340d1b6f135c72", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.7, + "rows": 8 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-bb511a1d2e44a7c206d09e43daec7af1d9a78b2e", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.12, + "rows": 1 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-dc0abc56f335aa048192b1ff173da6a036d54322", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.11, + "rows": 1 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-e29917d2c7eb818e4ffda664a932b942e2aee2b0", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 1.27, + "rows": 25 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-e52d913aad5f83cb6c35c6494f15d77c7bce72db", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.13, + "rows": 1 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-ec1711f1bd0ab146225bca3057a17ae7a4154828", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.25, + "rows": 4 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-ec2103f6375fadb68375fd782e4de6d3daf1fccb", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 1.04, + "rows": 15 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-f10bc2f0443d6ae6d49217a4c0e6271451fd0ce5", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.27, + "rows": 4 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-f10f5a83cc4070533bdbe2be4e476e2220165b9d", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.19, + "rows": 3 + }, + { + "id": "jenkinsci-job-import-plugin-ordinary-ffdeb8e959d01e145dd1c970595f929cb562f360", + "repo": "jenkinsci/job-import-plugin", + "kind": "ordinary", + "seconds": 0.14, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-5022afedadfe8449", + "repo": "jenkinsci/p4-plugin", + "kind": "vulnerability", + "seconds": 6.27, + "rows": 6 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-0cfd50a9fdc86cfe3e9a770038d273ea382f59bb", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 3.38, + "rows": 2 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-171dde20917615e63a6120251d7cb367f9d52992", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 2.0, + "rows": 3 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-18e4708722279a3a07eafb35f1419dfed6c7dbd1", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.37, + "rows": 2 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-1a38001c4dc8e2424479d44f95f6e25e25df4959", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 3.75, + "rows": 3 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-1d19356cb9f2d8e9b72a129e5719cca45d89821f", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.0, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-2ec2dccaa9e6ff318ccada47f2af691170aaa7be", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.83, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-30fb98f8e26adaa33762b526cdd443aa6122ac5b", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.06, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-46fb68bfea093c51f841bc6bdec9676bc4d468c7", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 10.1, + "rows": 14 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-4cb6dab4456e1afb3de1230cfacdce3e162fe8ea", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.8, + "rows": 2 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-65c18b225338e93837ef6882ad79cd9773e6e2c4", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 2.04, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-65fbed70350a4435ec091390d92722bbf7cc1c68", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.81, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-7b512b08c584a76d57f4401634f8e9736a365223", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 1.05, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-8a67359f1a69264da35c9e837273a649b3345b9d", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 0.83, + "rows": 1 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-a33e38e16ececd6e093eb213a9b8a3ab8352f937", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 28.32, + "rows": 58 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-aa910d56936a308600bdad568e2ab23ada636b7d", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 2.21, + "rows": 6 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-b2f7b1ba3eea1a81370dec019ebff29931875182", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 9.2, + "rows": 23 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-b42c75d6823b9e67bd968679494d1090f5d77941", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 2.31, + "rows": 5 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-dca0f55c69c487611689e72230e211ef4b760d9e", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 2.13, + "rows": 2 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-f4c24f7964bd1de847ace7b43546be81e51d2ccc", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 13.41, + "rows": 23 + }, + { + "id": "jenkinsci-p4-plugin-ordinary-f74fb378f0e7ce258aae449dc70f35820a1b11ed", + "repo": "jenkinsci/p4-plugin", + "kind": "ordinary", + "seconds": 11.4, + "rows": 12 + }, + { + "id": "shopperlabs-shopper-02254beed509364c", + "repo": "shopperlabs/shopper", + "kind": "vulnerability", + "seconds": 182.45, + "rows": 54 + }, + { + "id": "shopperlabs-shopper-ordinary-0db4faf747f7bc4d9eadacbfa0dcbd2620251ca5", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 1.14, + "rows": 1 + }, + { + "id": "shopperlabs-shopper-ordinary-13037bfd039904801dd8103dec3d0e720c547b94", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 5.82, + "rows": 2 + }, + { + "id": "shopperlabs-shopper-ordinary-17778cdd2c24a50d2fb29dae4a3aab5390b1b996", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 2.2, + "rows": 1 + }, + { + "id": "shopperlabs-shopper-ordinary-1aa9a4c2f471f00914a230b8168b3926f39837a7", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 1.4, + "rows": 1 + }, + { + "id": "shopperlabs-shopper-ordinary-31f33b1631254639836ce06ad6b078e6cfb0b13a", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 1.48, + "rows": 1 + }, + { + "id": "shopperlabs-shopper-ordinary-44809499ef406c9e199965fff6d124434857a36b", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 13.72, + "rows": 8 + }, + { + "id": "shopperlabs-shopper-ordinary-4628430c0a70506e15fba209557606f860566598", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 55.13, + "rows": 37 + }, + { + "id": "shopperlabs-shopper-ordinary-5394b647023599eb9d9a40f7b9ccc434cd8136a3", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 76.69, + "rows": 63 + }, + { + "id": "shopperlabs-shopper-ordinary-5a079da974a3d7f6e7ca0de3c6ca6b0849e304a2", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 30.55, + "rows": 3 + }, + { + "id": "shopperlabs-shopper-ordinary-7e56ccd817b0dc835919fac74db792e15ee62c1d", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 4.2, + "rows": 4 + }, + { + "id": "shopperlabs-shopper-ordinary-7eaab898afd90140f2162149585faf2642e18987", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 6.63, + "rows": 4 + }, + { + "id": "shopperlabs-shopper-ordinary-7f7e6bdb67509a50fae95f13a829494a8a3ab7b4", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 2.28, + "rows": 6 + }, + { + "id": "shopperlabs-shopper-ordinary-886e671270b21aed8ab851b5b82a53fc6e87fbc6", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 10.53, + "rows": 6 + }, + { + "id": "shopperlabs-shopper-ordinary-9313ed20032199988f571d7e655255819f7f6f04", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 11.97, + "rows": 10 + }, + { + "id": "shopperlabs-shopper-ordinary-a0c5bd3a05a0fef58b9043273d6062124a3ecf8e", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 48.67, + "rows": 29 + }, + { + "id": "shopperlabs-shopper-ordinary-a5a2760dc0c63872b55afec0e3d452df7989393b", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 2444.46, + "rows": 27 + }, + { + "id": "shopperlabs-shopper-ordinary-b178fe37da1c31f9f0057f35fdf4145936a25138", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 7.55, + "rows": 3 + }, + { + "id": "shopperlabs-shopper-ordinary-ceaea6a078025d4a7634d2646d27dde3f13b6f58", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 6.82, + "rows": 2 + }, + { + "id": "shopperlabs-shopper-ordinary-fada4250aaf3350c1db92698259c54f9ae6b4bb0", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 280.17, + "rows": 2 + }, + { + "id": "shopperlabs-shopper-ordinary-fcc3888151b9c43843c01f631870dfe7e8fd6329", + "repo": "shopperlabs/shopper", + "kind": "ordinary", + "seconds": 2.04, + "rows": 1 + }, + { + "id": "triggerdotdev-trigger.dev-0eb72665259b13c2", + "repo": "triggerdotdev/trigger.dev", + "kind": "vulnerability", + "seconds": 122.27, + "rows": 8 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-0e63f8317e7c8b472d8c9c7bb36ad3986d0238c7", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 91.01, + "rows": 19 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-22a7e33df74729c0a2dbd4b12d30c60378ec677f", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 25.82, + "rows": 2 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-2d569f64f8027bd2dde80ce14042542200f8fc7d", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 172.36, + "rows": 10 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-35298ac357d4717203cc181986d29109ae90e6cd", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 88.91, + "rows": 7 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-409388365eaaa9bb61eb4a9060e1fb46c8acc6a2", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 40.13, + "rows": 7 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-417ab876e3177ced4a4aa46c3de664b10950c6fe", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 46.5, + "rows": 7 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-478adb5d74d36f860f6f22d28ee18a299d692f45", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 41.79, + "rows": 3 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-4c42f6cc0b183a1435ec39d831892ca58b415008", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 357.99, + "rows": 33 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-55d85d0b2360cdeaecbf94f9fea8404cc5853acb", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 33.66, + "rows": 5 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-6be6c1816c2dd2dc09fc361f40be6af2a07c2de2", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 385.36, + "rows": 22 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-771937adf5d6b72943a006ec86b5bfe1057804e7", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 35.35, + "rows": 4 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-7ab437c8ad8ef1db66132157056cb59d94e763a8", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 741.52, + "rows": 39 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-87448ccaf23b7f99f505f953f2ae6fd6c3f3975c", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 27.53, + "rows": 3 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-9f29eb209042c4682f3fab90b4036dba300ea892", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 123.69, + "rows": 10 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-bfa902bd1822812e8f0aaf0d6831414dc70c13aa", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 354.28, + "rows": 45 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-cd518683c8c7830176bb1f9efdcbfe4b03d79457", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 13.76, + "rows": 1 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-cf4aa7e918eb6c3634b7dc8827b21be246519591", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 75.25, + "rows": 7 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-efcb89ac26984aac8e98f9c6ccb9a902c0483881", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 10.8, + "rows": 1 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-f78261a606362cf5480679e7286585367d744d08", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 327.86, + "rows": 18 + }, + { + "id": "triggerdotdev-trigger.dev-ordinary-fdec5a09a72fae962f3db9b3dcf8af08d34d06ad", + "repo": "triggerdotdev/trigger.dev", + "kind": "ordinary", + "seconds": 30.7, + "rows": 1 + } +] \ No newline at end of file diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/screen.csv b/benchmarks/measurements/2026-10-13-access-control-screen/screen.csv new file mode 100644 index 0000000..5ebfe98 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/screen.csv @@ -0,0 +1,9 @@ +annotation,strongest_feature,auc,n +callers,source.caller_count,0.7206525501054023,933 +change_summary,source.is_module,0.3577462794575717,933 +def_use,semantic.binds,0.35999015748031493,262 +entry_points,source.entry_point_count,0.7070918415505144,933 +function_text,source.span_lines,0.36404681748104334,933 +hunks,diff.deletion_count,0.8854275687988672,952 +obligations,semantic.obligations_undischarged,0.7620570866141732,262 +roles,semantic.sinks,0.17441860465116277,46 diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/separation.csv b/benchmarks/measurements/2026-10-13-access-control-screen/separation.csv new file mode 100644 index 0000000..87ed5bb --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/separation.csv @@ -0,0 +1,18 @@ +column,auc,n,note,annotation +diff.distance,0.7519618922057947,935,,hunks +diff.on_changed_line,0.26616080773256173,952,,hunks +diff.hunk_count,0.26391368589546266,952,,hunks +diff.hunk_lines,0.1529581973773318,952,,hunks +diff.deletion_count,0.8854275687988672,952,,hunks +source.span_lines,0.36404681748104334,933,,function_text +source.is_module,0.3577462794575717,933,,change_summary +source.caller_count,0.7206525501054023,933,,callers +source.caller_files,0.7166567032690432,933,,callers +source.entry_point_count,0.7070918415505144,933,,entry_points +semantic.sources,0.6976744186046512,46,,roles +semantic.sinks,0.17441860465116277,46,,roles +semantic.sanitizers,0.4534883720930233,46,,roles +semantic.binds,0.35999015748031493,262,,def_use +semantic.calls,0.5642224409448819,262,,def_use +semantic.obligations,0.7541830708661417,262,,obligations +semantic.obligations_undischarged,0.7620570866141732,262,,obligations diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/separation_deletion_rows.csv b/benchmarks/measurements/2026-10-13-access-control-screen/separation_deletion_rows.csv new file mode 100644 index 0000000..60e069a --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/separation_deletion_rows.csv @@ -0,0 +1,18 @@ +column,auc,n,note,annotation +diff.distance,0.7289761520530752,235,,hunks +diff.on_changed_line,0.31410256410256415,235,,hunks +diff.hunk_count,0.1386049847588309,235,,hunks +diff.hunk_lines,0.12058454366146672,235,,hunks +diff.deletion_count,0.708086785009862,235,,hunks +source.span_lines,0.3603639949793796,235,,function_text +source.is_module,0.41420118343195267,235,,change_summary +source.caller_count,0.7297382105074413,235,,callers +source.caller_files,0.7217141832526447,235,,callers +source.entry_point_count,0.8280437511206741,235,,entry_points +semantic.sources,,7,one class or empty,roles +semantic.sinks,,7,one class or empty,roles +semantic.sanitizers,,7,one class or empty,roles +semantic.binds,,24,one class or empty,def_use +semantic.calls,,24,one class or empty,def_use +semantic.obligations,,24,one class or empty,obligations +semantic.obligations_undischarged,,24,one class or empty,obligations diff --git a/benchmarks/measurements/2026-10-13-access-control-screen/summary.json b/benchmarks/measurements/2026-10-13-access-control-screen/summary.json new file mode 100644 index 0000000..c2aa138 --- /dev/null +++ b/benchmarks/measurements/2026-10-13-access-control-screen/summary.json @@ -0,0 +1,24 @@ +{ + "producer_rows_ok": { + "diff": 952, + "source": 933, + "semantic": 262, + "taxonomy": 952 + }, + "table_digest": "02774edc4c07dcc92b2fcf04ed342e7d0bc8afe761c808fef2d11b473bd99271", + "rows": 952, + "labelled": { + "positive": 74, + "negative": 878 + }, + "repositories": 5, + "changes": 105, + "positive_repositories": 5, + "source_failures": { + "FileNotFoundError": 16, + "ValueError": 3 + }, + "unread_rows": 0, + "positive_rows_on_changed_line": 12, + "positive_rows_with_deletions": 66 +} \ No newline at end of file