Open_Ultra_SAST/benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md

Valid detection benchmark, pilot (2026-10-12)

First build of the benchmark defined in .kiro/specs/valid-detection-benchmark/. Labels come from the fix, not SZZ. The frozen manifest is at benchmarks/valid/pilot/ (pool-valid.toml, freeze-valid.json). This directory keeps the provenance.

What was built

Step	Result
Sourcing	32 held-out Python/JS advisories taken from the public part of pool-p1 (pip 12, npm 20). CWE and advisory text fetched from OSV by GHSA id. No fresh GitHub harvest.
Fix-site catalog	133 candidate locations across 31 repos, labelled at fix^1 (the commit just before the fix). 125 after dropping 7 test files and 1 minified file. method = "fix_site" on every row; the SZZ methods were never called.
LLM verification	125 DeepSeek calls (deepseek-flash). 31 locations passed all three checks (vulnerable before the fix, fix addresses it, reachable from an in-repo entry point) across 12 repos. 94 rejected.
Human spot-check	All 12 repos accepted by the maintainer (reviewer mc, 2026-10-12), recorded on the 31 locations.
Negatives	20 ordinary first-parent commits per positive change, drawn from the same repo, fix excluded. 240 in total.
Held-out guard	0 dropped against the training-only used set (1777 names: pairs, pointers, harvest, units, sweep).
Freeze	252 changes: 12 vulnerability + 240 ordinary. Digest in freeze-valid.json.

The 12 positives

Repo	Lang	Family	Advisory	Vulnerable function
bytebase/dbhub	ts	access_control	GHSA-mwwr-p57h-56pf	src/utils/sql-parser.ts::scanTokenMySQL
contentful/contentful-mcp-server	ts	untrusted_destination	GHSA-2xhg-73j7-rrgx	.../exportSpace.ts::tool (+1)
fastapi-users/fastapi-users	py	access_control	GHSA-5j53-63w8-8625	fastapi_users/router/oauth.py::get_oauth_router (+2)
i18next/i18next-http-middleware	js	output_encoding	GHSA-c3h8-g69v-pjrg	lib/index.js::extendOptionsWithDefaults
imzbf/md-editor-v3	ts	output_encoding	GHSA-3rm2-h79c-8qw6	.../useMarkdownIt.ts::useMarkdownIt (+1)
jupyter-lsp/jupyterlab-lsp	py	path	GHSA-4qhp-652w-c22x	.../virtual_documents_shadow.py::shadow_virtual_documents
matrix-org/matrix-appservice-bridge	ts	access_control	GHSA-35g4-qx3c-vjhx	src/components/room-upgrade-handler.ts::onJoinedNewRoom
nvbn/thefuck	py	path	GHSA-8wwf-2644-f8x4	thefuck/rules/dirty_untar.py::side_effect (+1)
ondata/ckan-mcp-server	ts	untrusted_destination	GHSA-798p-78g2-v556	src/utils/http.ts::validateServerUrl (+1)
webpack/webpack-bundle-analyzer	js	output_encoding	GHSA-pgr8-jg6h-8gw6	src/viewer.js::chartData
zenitram/malla	js	output_encoding	GHSA-ch57-39q2-4crm	src/malla/static/js/chat.js::refreshPop (+12)
zereight/gitlab-mcp	ts	path	GHSA-cv3r-c5h8-f4g5	index.ts::startSSEServer (+1)

"(+N)" means the fix touched N more functions the verifier also accepted. They collapse into one positive change per fix commit in the manifest; the extra functions are listed as targets on that change.

Decisions made while building

• Parent is fix^1, read from git, not the parent field stored in pool-p1. That field did not always equal the fix commit's real first parent. "The code just before the fix" is fix^1.
• Candidates are every function the fix touched. fix_sites() does not know which function holds the vulnerability, so the catalog over-generates and the LLM check plus the human spot-check select the real location. One repo (zenitram/malla) produced 40 candidates from one fix.
• The negative ratio is per positive change, not per accepted function. assemble.py was corrected; the old count would have drawn 260 negatives for malla alone.
• The held-out guard runs against training sources only. build_used() also scans measurements/ and populations/, which contain the SZZ pool and analysis of it, so it flagged all 12 repos as "used" by my own earlier measurements. That is circular. None of the 12 appear in any training source (pairs, pointers, harvest, units, sweep).
• Catalog enumeration skips a site it cannot read (fix renamed or removed the function) instead of aborting the whole run. Verification stays strict.

Not done yet

• Replay (whole tool, all tiers on) has not run. 252 changes at the engine's ~200 s fixed cost is about 14 hours one container at a time on the local host; that scale belongs on the kube ax lane, still blocked on the gVisor checkpoint path.
• Metrics (recall, precision, false alarms per 100 pushes, exploit-verified rate) follow the replay.
• Growing from 12 to the ~40 positives score.py:required_n asks for.

Reproduce

# clones: fetch each repo single-branch with history, then fetch <fix> <parent>
python -m benchmarks.valid.catalog  sources.json --repos repos.json --out catalog.toml
DEEPSEEK_API_KEY=... python -m benchmarks.valid.verify run catalog.toml --repos repos.json --out verification.json
python -m benchmarks.valid.verify spot-check verification.json <id> accept --reviewer mc --date 2026-10-12
python - <<'PY'   # assemble with the training-only used set
from benchmarks.valid import assemble
...  # see this directory's provenance files; used = build_used(root) minus {measurements, populations}
PY