feat(benchmark): freeze the valid-benchmark pilot, 12 verified positives + 240 negatives

First live run of the valid detection benchmark. Labels come from the fix site (fix^1), never SZZ. - 32 held-out Python/JS advisories sourced from the public pool-p1 set, CWE + text from OSV. 133 candidate locations -> 125 after dropping tests/minified. - 125 DeepSeek verification calls: 31 locations passed all three checks (vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos. - Maintainer spot-check accepted all 12. - 20 ordinary negatives per positive change from the same repos, 240 total. - Held-out guard: 0 dropped against the training-only used set. - Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/. Three fixes found by the live run: - catalog.py: skip a site the fix renamed or removed instead of aborting. - assemble.py: negative ratio counts positive changes, not accepted functions (malla would have demanded 260 negatives). Test updated. - held-out guard: run against training sources only; build_used() also scans measurements/populations, which flagged the 12 repos via my own SZZ-pool analysis outputs (circular). Replay and metrics not yet run; see NOTES. Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-16 03:49:26 +00:00 · 2026-10-12 19:42:45 +02:00
parent 43ef254134
commit 7a636d60fb
10 changed files with 4606 additions and 3 deletions
@@ -0,0 +1,75 @@
+# Valid detection benchmark, pilot (2026-10-12)
+
+First build of the benchmark defined in `.kiro/specs/valid-detection-benchmark/`. Labels come from
+the fix, not SZZ. The frozen manifest is at `benchmarks/valid/pilot/` (`pool-valid.toml`,
+`freeze-valid.json`). This directory keeps the provenance.
+
+## What was built
+
+| Step | Result |
+|---|---|
+| Sourcing | 32 held-out Python/JS advisories taken from the public part of `pool-p1` (pip 12, npm 20). CWE and advisory text fetched from OSV by GHSA id. No fresh GitHub harvest. |
+| Fix-site catalog | 133 candidate locations across 31 repos, labelled at `fix^1` (the commit just before the fix). 125 after dropping 7 test files and 1 minified file. `method = "fix_site"` on every row; the SZZ methods were never called. |
+| LLM verification | 125 DeepSeek calls (`deepseek-flash`). 31 locations passed all three checks (vulnerable before the fix, fix addresses it, reachable from an in-repo entry point) across 12 repos. 94 rejected. |
+| Human spot-check | All 12 repos accepted by the maintainer (reviewer `mc`, 2026-10-12), recorded on the 31 locations. |
+| Negatives | 20 ordinary first-parent commits per positive change, drawn from the same repo, fix excluded. 240 in total. |
+| Held-out guard | 0 dropped against the training-only used set (1777 names: pairs, pointers, harvest, units, sweep). |
+| Freeze | 252 changes: 12 `vulnerability` + 240 `ordinary`. Digest in `freeze-valid.json`. |
+
+## The 12 positives
+
+| Repo | Lang | Family | Advisory | Vulnerable function |
+|---|---|---|---|---|
+| bytebase/dbhub | ts | access_control | GHSA-mwwr-p57h-56pf | `src/utils/sql-parser.ts::scanTokenMySQL` |
+| contentful/contentful-mcp-server | ts | untrusted_destination | GHSA-2xhg-73j7-rrgx | `.../exportSpace.ts::tool` (+1) |
+| fastapi-users/fastapi-users | py | access_control | GHSA-5j53-63w8-8625 | `fastapi_users/router/oauth.py::get_oauth_router` (+2) |
+| i18next/i18next-http-middleware | js | output_encoding | GHSA-c3h8-g69v-pjrg | `lib/index.js::extendOptionsWithDefaults` |
+| imzbf/md-editor-v3 | ts | output_encoding | GHSA-3rm2-h79c-8qw6 | `.../useMarkdownIt.ts::useMarkdownIt` (+1) |
+| jupyter-lsp/jupyterlab-lsp | py | path | GHSA-4qhp-652w-c22x | `.../virtual_documents_shadow.py::shadow_virtual_documents` |
+| matrix-org/matrix-appservice-bridge | ts | access_control | GHSA-35g4-qx3c-vjhx | `src/components/room-upgrade-handler.ts::onJoinedNewRoom` |
+| nvbn/thefuck | py | path | GHSA-8wwf-2644-f8x4 | `thefuck/rules/dirty_untar.py::side_effect` (+1) |
+| ondata/ckan-mcp-server | ts | untrusted_destination | GHSA-798p-78g2-v556 | `src/utils/http.ts::validateServerUrl` (+1) |
+| webpack/webpack-bundle-analyzer | js | output_encoding | GHSA-pgr8-jg6h-8gw6 | `src/viewer.js::chartData` |
+| zenitram/malla | js | output_encoding | GHSA-ch57-39q2-4crm | `src/malla/static/js/chat.js::refreshPop` (+12) |
+| zereight/gitlab-mcp | ts | path | GHSA-cv3r-c5h8-f4g5 | `index.ts::startSSEServer` (+1) |
+
+"(+N)" means the fix touched N more functions the verifier also accepted. They collapse into one
+positive change per fix commit in the manifest; the extra functions are listed as `targets` on that
+change.
+
+## Decisions made while building
+
+- **Parent is `fix^1`, read from git**, not the `parent` field stored in `pool-p1`. That field did
+  not always equal the fix commit's real first parent. "The code just before the fix" is `fix^1`.
+- **Candidates are every function the fix touched.** `fix_sites()` does not know which function
+  holds the vulnerability, so the catalog over-generates and the LLM check plus the human spot-check
+  select the real location. One repo (`zenitram/malla`) produced 40 candidates from one fix.
+- **The negative ratio is per positive change**, not per accepted function. `assemble.py` was
+  corrected; the old count would have drawn 260 negatives for malla alone.
+- **The held-out guard runs against training sources only.** `build_used()` also scans
+  `measurements/` and `populations/`, which contain the SZZ pool and analysis of it, so it flagged all
+  12 repos as "used" by my own earlier measurements. That is circular. None of the 12 appear in any
+  training source (pairs, pointers, harvest, units, sweep).
+- **Catalog enumeration skips a site it cannot read** (fix renamed or removed the function) instead
+  of aborting the whole run. Verification stays strict.
+
+## Not done yet
+
+- **Replay** (whole tool, all tiers on) has not run. 252 changes at the engine's ~200 s fixed cost is
+  about 14 hours one container at a time on the local host; that scale belongs on the kube `ax` lane,
+  still blocked on the gVisor checkpoint path.
+- **Metrics** (recall, precision, false alarms per 100 pushes, exploit-verified rate) follow the replay.
+- Growing from 12 to the ~40 positives `score.py:required_n` asks for.
+
+## Reproduce
+
+```
+# clones: fetch each repo single-branch with history, then fetch <fix> <parent>
+python -m benchmarks.valid.catalog  sources.json --repos repos.json --out catalog.toml
+DEEPSEEK_API_KEY=... python -m benchmarks.valid.verify run catalog.toml --repos repos.json --out verification.json
+python -m benchmarks.valid.verify spot-check verification.json <id> accept --reviewer mc --date 2026-10-12
+python - <<'PY'   # assemble with the training-only used set
+from benchmarks.valid import assemble
+...  # see this directory's provenance files; used = build_used(root) minus {measurements, populations}
+PY
+```
@@ -0,0 +1 @@
+["bytebase/dbhub", "contentful/contentful-mcp-server", "fastapi-users/fastapi-users", "i18next/i18next-http-middleware", "imzbf/md-editor-v3", "jupyter-lsp/jupyterlab-lsp", "matrix-org/matrix-appservice-bridge", "nvbn/thefuck", "ondata/ckan-mcp-server", "webpack/webpack-bundle-analyzer", "zenitram/malla", "zereight/gitlab-mcp"]
@@ -77,7 +77,11 @@ def freeze(
        head = (history_heads or {}).get(repo) or git.run("rev-parse", "HEAD").strip()
        heads[repo] = head
        history = sorted(extract.ordinary_history(git, head), key=lambda row: row["head"])
-        ordinary, exclusions = negatives(history, fixes, functions, count=ratio * len(positives), seed=seed)
+        # Ratio is per positive CHANGE (a fix commit), not per accepted function. Several
+        # accepted functions in one fix collapse to one vulnerability change below, so the
+        # negative count must track distinct (commit, parent) pairs.
+        change_count = len({(case["commit"], case["parent"]) for case in positives})
+        ordinary, exclusions = negatives(history, fixes, functions, count=ratio * change_count, seed=seed)
        draws[repo] = exclusions
        changes = []
        pairs = defaultdict(list)
@@ -103,7 +103,13 @@ def build_cases(git, source, *, cwe: str | None = None, advisory_text: str = "")
            function=site["function"],
            method="fix_site",
        )
+        try:
            sides = read_functions(git, case)
+        except extract.InstrumentFailure:
+            # The fix renamed, moved, or removed the function, so there is no clean
+            # pre/post pair at this site. Skip it during enumeration. Verification
+            # stays strict: a verdict on an unread case is still rejected there.
+            continue
        if extract.normalized(sides["pre_fix"]) == extract.normalized(sides["post_fix"]):
            continue
        case["function_lines"] = [sides["pre_fix_lines"]]
@@ -0,0 +1,165 @@
+{
+  "catalog_digest": "3f641f15619a54351ae5032fb06cc97f37d5876697f3b71bb966a3c1f539d3e4",
+  "dropped_count": 0,
+  "freeze_digest": "c866c0e812c0b54476c7168fdb0e4a20652de0d30485270afecbe51e5c35e424",
+  "guard_result": 0,
+  "history_heads": {
+    "bytebase/dbhub": "f7b31468be4ff0c70b1c662b3e535fe36f4dad06",
+    "contentful/contentful-mcp-server": "edb981b6e2a75173ff57e62ca344bc76a5eec766",
+    "fastapi-users/fastapi-users": "e9c57ff7ff2fd385bb0f6f795b524e9736b64290",
+    "i18next/i18next-http-middleware": "824bdfb49e777a499a13ea19fa70a07f8fe68ad2",
+    "imzbf/md-editor-v3": "1a6f4c61ebf38a836cc985b786f39a126441a835",
+    "jupyter-lsp/jupyterlab-lsp": "e2e569321bc45702718030fabfe7dd38d0457f25",
+    "matrix-org/matrix-appservice-bridge": "6392ab0550e23fffa9f014411148c667ba2e77da",
+    "nvbn/thefuck": "c7e7e1d884d3bb241ea6448f72a989434c2a35ec",
+    "ondata/ckan-mcp-server": "d7a24143699c64c4dcea0ca7041a19fc875f8cef",
+    "webpack/webpack-bundle-analyzer": "6046503c6e183cc2bcf069a7d58dcf87f4870f85",
+    "zenitram/malla": "97d141888677008e856a833b68199169efa6c828",
+    "zereight/gitlab-mcp": "3a5dff69ed7df5a0d1c3af388714f188ff2dc4af"
+  },
+  "label_check_counts": {
+    "confirmed": 31,
+    "refactor": 0,
+    "unchecked": 0,
+    "unclear": 0
+  },
+  "negative_ratio": 20,
+  "ordinary_exclusions": {
+    "bytebase/dbhub": {
+      "advisory_fix": 1,
+      "dependency": 13,
+      "eligible": 273,
+      "manifest_only": 54,
+      "merge_or_root": 19,
+      "no_source": 210,
+      "revert": 3,
+      "security": 26,
+      "size_cap": 12,
+      "vulnerable_function": 3
+    },
+    "contentful/contentful-mcp-server": {
+      "advisory_fix": 1,
+      "dependency": 202,
+      "eligible": 53,
+      "manifest_only": 1,
+      "merge_or_root": 14,
+      "no_source": 41,
+      "revert": 2,
+      "security": 12,
+      "size_cap": 10,
+      "vulnerable_function": 10
+    },
+    "fastapi-users/fastapi-users": {
+      "advisory_fix": 1,
+      "dependency": 346,
+      "eligible": 171,
+      "manifest_only": 18,
+      "merge_or_root": 19,
+      "no_source": 154,
+      "revert": 4,
+      "security": 17,
+      "size_cap": 2,
+      "vulnerable_function": 31
+    },
+    "i18next/i18next-http-middleware": {
+      "advisory_fix": 1,
+      "dependency": 47,
+      "eligible": 70,
+      "manifest_only": 74,
+      "merge_or_root": 4,
+      "no_source": 43,
+      "security": 8,
+      "vulnerable_function": 15
+    },
+    "imzbf/md-editor-v3": {
+      "advisory_fix": 1,
+      "dependency": 7,
+      "eligible": 576,
+      "manifest_only": 226,
+      "merge_or_root": 100,
+      "no_source": 161,
+      "revert": 7,
+      "security": 12,
+      "size_cap": 6,
+      "vulnerable_function": 51
+    },
+    "jupyter-lsp/jupyterlab-lsp": {
+      "advisory_fix": 1,
+      "dependency": 36,
+      "eligible": 131,
+      "manifest_only": 19,
+      "merge_or_root": 383,
+      "no_source": 90,
+      "size_cap": 5,
+      "vulnerable_function": 2
+    },
+    "matrix-org/matrix-appservice-bridge": {
+      "dependency": 18,
+      "eligible": 172,
+      "manifest_only": 10,
+      "merge_or_root": 235,
+      "no_source": 141,
+      "security": 4,
+      "size_cap": 3
+    },
+    "nvbn/thefuck": {
+      "advisory_fix": 1,
+      "dependency": 76,
+      "eligible": 429,
+      "manifest_only": 10,
+      "merge_or_root": 299,
+      "no_source": 114,
+      "revert": 3,
+      "security": 4,
+      "vulnerable_function": 1
+    },
+    "ondata/ckan-mcp-server": {
+      "advisory_fix": 1,
+      "dependency": 681,
+      "eligible": 72,
+      "manifest_only": 4,
+      "merge_or_root": 10,
+      "no_source": 310,
+      "revert": 2,
+      "security": 39,
+      "size_cap": 5,
+      "vulnerable_function": 71
+    },
+    "webpack/webpack-bundle-analyzer": {
+      "dependency": 43,
+      "eligible": 82,
+      "manifest_only": 18,
+      "merge_or_root": 104,
+      "no_source": 131,
+      "revert": 2,
+      "security": 4,
+      "size_cap": 1,
+      "vulnerable_function": 18
+    },
+    "zenitram/malla": {
+      "advisory_fix": 1,
+      "dependency": 5,
+      "eligible": 76,
+      "manifest_only": 1,
+      "merge_or_root": 1,
+      "no_source": 25,
+      "security": 8,
+      "size_cap": 3,
+      "vulnerable_function": 7
+    },
+    "zereight/gitlab-mcp": {
+      "advisory_fix": 1,
+      "dependency": 77,
+      "eligible": 170,
+      "manifest_only": 41,
+      "merge_or_root": 280,
+      "no_source": 211,
+      "revert": 3,
+      "security": 16,
+      "size_cap": 7,
+      "vulnerable_function": 8
+    }
+  },
+  "used_set_digest": "fb7b0cc1f38bc2fb513123d8b4a0e16afd5501b42ef1a49ad7ecedd8eba55885",
+  "verification_digest": "9363de4b7466953ad72cb29f3e91a01b3cfc8411b35310e0fa270f967836b41b"
+}
@@ -438,7 +438,9 @@ def test_multi_function_fix_is_one_push_with_all_case_ids(pair, tmp_path, monkey
        tmp_path / "multi", cases=cases, verifications=[accepted(case) for case in cases], repos={original["repo"]: git}, used=set()
    )
    loaded = replay.load_slice(tmp_path / "multi/pool-valid.toml", 1)
-    assert len(loaded) == 41 and len({row["id"] for row in loaded}) == 41
+    # One fix commit is one positive change, so the 20x negative ratio is 20, not 20 per
+    # accepted function. Two accepted functions still collapse to a single vulnerability push.
+    assert len(loaded) == 21 and len({row["id"] for row in loaded}) == 21
    (positive,) = [row for row in loaded if row["kind"] == "vulnerability"]
    assert set(positive["case_ids"]) == {case["id"] for case in cases}
    target = positive["targets"][1]
				`@@ -0,0 +1 @@`
				`["bytebase/dbhub", "contentful/contentful-mcp-server", "fastapi-users/fastapi-users", "i18next/i18next-http-middleware", "imzbf/md-editor-v3", "jupyter-lsp/jupyterlab-lsp", "matrix-org/matrix-appservice-bridge", "nvbn/thefuck", "ondata/ckan-mcp-server", "webpack/webpack-bundle-analyzer", "zenitram/malla", "zereight/gitlab-mcp"]`