Open_Ultra_SAST/benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md

# Valid detection benchmark, pilot (2026-10-12)

First build and first run of the benchmark defined in `.kiro/specs/valid-detection-benchmark/`.
Labels come from the fix, not SZZ. The frozen manifest is at `benchmarks/valid/pilot/`
(`pool-valid.toml`, `freeze-valid.json`). This directory keeps the provenance and the numbers.

## Numbers (read 2026-10-13 morning)

Two arms over the same 252 changes (12 verified positives, 240 ordinary pushes from the same 12
repos), as R6 of the requirements asks:

- **(a) push hook as shipped**: quick rules + engine, 300 s deadline, the Docker replay lane, no
  model key in the container. Finished 2026-10-13 15:38: 252 of 252 records (12 positives, 240
  ordinary).
- **(b) hunter arm**: the LLM tool hunter (`scan --mode deep`'s search) run off-cluster over the
  changed files at head, every finding classified into the taxonomy by the decision-engine
  classifier. Finished: 252 of 252. Reported twice: raw (every hunter suspicion counts) and
  filtered (only suspicions the classifier judged `vulnerable`).

| Metric | (a) push hook | (b) hunter, raw | (b') hunter, classifier-filtered | (a) + (b') whole tool |
|---|---|---|---|---|
| Recall (12 positives) | 0 / 12 = 0% (Wilson 0 to 24.3) | 6 / 12 = 50% (25.4 to 74.6) | 5 / 12 = 41.7% (19.3 to 68.0) | 5 / 12 = 41.7% (19.3 to 68.0) |
| False alarms per 100 ordinary pushes | 5 / 238 = 2.1 (0.9 to 4.8) | 221 / 239 = 92.5 (88.4 to 95.2) | 130 / 239 = 54.4 (48.1 to 60.6) | 132 / 238 = 55.5 (49.1 to 61.6) |
| Precision (flagged pushes that were positives) | 0 / 5 | 6 / 232 = 2.6% | 5 / 139 = 3.6% | 5 / 141 = 3.5% |
| Exploit-verified | 0 (no proof run) | 0 (no proof run) | 0 (no proof run) | 0 (no proof run) |
| Instrument failures | 2 / 252 | 1 / 252 | 1 / 252 | 2 / 252 |

Neither arm meets the gate (recall at least 90%, false alarms under 10 per 100 pushes). The hook as
shipped is quiet and blind on this set. The hunter finds about half and alarms on more than half of
ordinary pushes even after the classifier. The whole-tool number the brief holds the tool to is
(a) plus (b'): 41.7% recall at 55.5 false alarms per 100 pushes. (a) caught nothing, so recall equals (b'); of its five false alarms, two were on pushes (b') had already flagged and three were new. The denominator is 238, not 239, because (a) has two instrument failures among the ordinary pushes.

Files: `scored_a.json`, `metrics_a.json`, `scored_combined.json`, `metrics_combined.json`, `scored_b_raw.json`,
`metrics_b_raw.json`, `scored_b_filtered.json`, `metrics_b_filtered.json`. Scorer:
`benchmarks/unseen/score.py` (a catch needs the labelled family and a location inside the labelled
function or within 10 lines of it). Metrics: `benchmarks/valid/metrics.py`.

### Per positive

| Repo | Label | (a) hook | (b) raw | (b') filtered | What the hunter did |
|---|---|---|---|---|---|
| bytebase/dbhub | access_control | miss (engine ran) | miss | miss | 8 suspicions in the right files, all `not_vulnerable` or `unsure`, none typed access_control |
| contentful/contentful-mcp-server | untrusted_destination | miss | miss | miss | 2 `vulnerable` hits in the labelled file, typed `path`; family mismatch, not credited |
| fastapi-users/fastapi-users | access_control | miss | miss | miss | 0 suspicions |
| i18next/i18next-http-middleware | output_encoding | miss | hit | miss | hit at `lib/index.js:96`, classifier said `unsure`, dropped by the filter |
| imzbf/md-editor-v3 | output_encoding | miss (engine ran) | hit | hit | 9 `vulnerable` html-injection suspicions at the labelled sites |
| jupyter-lsp/jupyterlab-lsp | path | miss | hit | hit | 3 `vulnerable` path hits in the labelled file |
| matrix-org/matrix-appservice-bridge | access_control | miss (engine ran) | miss | miss | 5 "complexity hotspot" suspicions in the right file, all `not_vulnerable` |
| nvbn/thefuck | path | miss (engine ran) | hit | hit | both labelled functions hit, `vulnerable` 0.85 to 0.9 |
| ondata/ckan-mcp-server | untrusted_destination | miss | hit | hit | 2 `vulnerable` ssrf hits in `http.ts` |
| webpack/webpack-bundle-analyzer | output_encoding | miss | miss | miss | `vulnerable` output_encoding hit in `viewer.js:185`, outside the labelled function |
| zenitram/malla | output_encoding | miss | hit | hit | 7 `vulnerable` dom-xss hits |
| zereight/gitlab-mcp | path | miss | miss | miss | 5 suspicions, none typed path |

Two misses are scorer strictness rather than blindness: contentful (right file, family typed `path`
instead of `untrusted_destination`) and webpack (right file and family, other function). Keep the
strict scorer; note them.

### Arm (a) detail

- Engine finished inside 300 s on 116 of 252 changes (46%); 134 fell back to quick rules only; 2
  instrument failures (`contentful` d5dac236, push artifact unread; `matrix` fed2acfb, no changed
  bytes read). On the positives: engine ran on 4 of 12, quick only on 8.
- Every record proves the instrument: hook exit 0, bytes read per changed file, base and head
  verified.
- All five false alarms are quick rules; the engine raised none:
  - `bytebase/dbhub` c10e1269, `js-command-exec` (CWE-78) at `src/connectors/sqlite/index.ts:132`
  - `nvbn/thefuck` 6e325eb7, `python-os-command` (CWE-78) at `thefuck/entrypoints/shell_logger.py:35`
  - `zenitram/malla` dea8d513 and 61c4fd5e, `python-sql-injection` (CWE-89) in the repository and analytics modules
  - `zereight/gitlab-mcp` 3d01ea54, `js-command-exec` (CWE-78) at `scripts/generate-tools-readme.ts:28`

### Arm (b) detail

- 1398 suspicions over 251 pushes: mean 5.6 per ordinary push, median 5, max 23.
- Classifier verdicts: 377 `vulnerable`, 968 `not_vulnerable`, 53 `unsure`, 0 errors. Families among
  the `vulnerable` ones: config_secrets 103, injection 74, output_encoding 58, access_control 54,
  path 39, untrusted_destination 33, prototype 10, other 6.
- The hunter's own `family:` tag is free text (`sql-parser`, `complexity-hotspot`, `dom-xss`), so
  without the classifier every finding scored `unknown`. Product follow-up: constrain the hunter
  prompt to the taxonomy (`tool_hunter.py`) so the push path emits taxonomy families natively.
- The instrument failure: `matrix` 7018468e, no readable hotspot (`hook_bytes_read: 0`).

### Where the hunter's false alarms come from (sweep over the same records, no new model calls)

Of the 343 ordinary-push suspicions the classifier called `vulnerable`, only 40 sit on a line the
push changed and only 93 within 10 lines of one. The hunter searches the whole changed file, so on
an ordinary push it re-reports the repo's pre-existing code. Scoping to the change is the biggest
lever, confidence the second, the `config_secrets` family the third. In-sample on 12 positives, so
treat the table as direction, not a tuned threshold:

| Filter on hunter suspicions | Recall | False alarms per 100 |
|---|---|---|
| raw | 6/12 | 92.5 |
| classifier `vulnerable` | 5/12 | 54.4 |
| `vulnerable`, confidence >= 0.8 | 5/12 | 31.8 |
| `vulnerable`, within 10 lines of the diff | 5/12 | 22.6 |
| `vulnerable`, confidence >= 0.8, within 10 lines | 5/12 | 11.3 |
| `vulnerable`, confidence >= 0.8, within 10 lines, not `config_secrets` | 5/12 | 8.8 |

The exact-line scope (0 lines) drops recall to 1/12 because the benchmark's "head" is the code
before the fix, where most fixes add lines rather than change them; a developer's own push has the
vulnerable lines in the diff by construction. Proof stays the last step, on what survives this.

### What to do with this

1. The push hook needs the LLM tier (owned by `pre-push-safety-net`): the hook as shipped caught
   none of the 12. That is the product gap R6 names.
2. The hunter needs a precision step before it can sit on pre-push: 54 alarms per 100 pushes after
   the classifier is far from 10. Candidates: require `vulnerable` with confidence at or above 0.7,
   drop `config_secrets` noise in ordinary pushes, restrict to changed lines, or run the proof arm on
   the survivors.
3. Twelve positives give intervals 50 points wide. Grow to the ~40 positives `score.py:required_n`
   asks for before quoting a single recall number.

## What was built

| Step | Result |
|---|---|
| Sourcing | 32 held-out Python/JS advisories taken from the public part of `pool-p1` (pip 12, npm 20). CWE and advisory text fetched from OSV by GHSA id. No fresh GitHub harvest. |
| Fix-site catalog | 133 candidate locations across 31 repos, labelled at `fix^1` (the commit just before the fix). 125 after dropping 7 test files and 1 minified file. `method = "fix_site"` on every row; the SZZ methods were never called. |
| LLM verification | 125 DeepSeek calls (`deepseek-flash`). 31 locations passed all three checks (vulnerable before the fix, fix addresses it, reachable from an in-repo entry point) across 12 repos. 94 rejected. |
| Human spot-check | All 12 repos accepted by the maintainer (reviewer `mc`, 2026-10-12), recorded on the 31 locations. |
| Negatives | 20 ordinary first-parent commits per positive change, drawn from the same repo, fix excluded. 240 in total. |
| Held-out guard | 0 dropped against the training-only used set (1777 names: pairs, pointers, harvest, units, sweep). |
| Freeze | 252 changes: 12 `vulnerability` + 240 `ordinary`. Digest in `freeze-valid.json`. |
| Replay, arm (a) | `benchmarks/unseen/replay.py --arm long_deadline`, Docker lane, one container at a time on the local host, about 14 records per hour. Started 2026-10-12 19:51. |
| Hunter, arm (b) | `benchmarks/valid/hunter_arm.py run`, positives 20:23 to 20:51, ordinary 20:51 to 22:00. |

## The 12 positives

| Repo | Lang | Family | Advisory | Vulnerable function |
|---|---|---|---|---|
| bytebase/dbhub | ts | access_control | GHSA-mwwr-p57h-56pf | `src/utils/sql-parser.ts::scanTokenMySQL` |
| contentful/contentful-mcp-server | ts | untrusted_destination | GHSA-2xhg-73j7-rrgx | `.../exportSpace.ts::tool` (+1) |
| fastapi-users/fastapi-users | py | access_control | GHSA-5j53-63w8-8625 | `fastapi_users/router/oauth.py::get_oauth_router` (+2) |
| i18next/i18next-http-middleware | js | output_encoding | GHSA-c3h8-g69v-pjrg | `lib/index.js::extendOptionsWithDefaults` |
| imzbf/md-editor-v3 | ts | output_encoding | GHSA-3rm2-h79c-8qw6 | `.../useMarkdownIt.ts::useMarkdownIt` (+1) |
| jupyter-lsp/jupyterlab-lsp | py | path | GHSA-4qhp-652w-c22x | `.../virtual_documents_shadow.py::shadow_virtual_documents` |
| matrix-org/matrix-appservice-bridge | ts | access_control | GHSA-35g4-qx3c-vjhx | `src/components/room-upgrade-handler.ts::onJoinedNewRoom` |
| nvbn/thefuck | py | path | GHSA-8wwf-2644-f8x4 | `thefuck/rules/dirty_untar.py::side_effect` (+1) |
| ondata/ckan-mcp-server | ts | untrusted_destination | GHSA-798p-78g2-v556 | `src/utils/http.ts::validateServerUrl` (+1) |
| webpack/webpack-bundle-analyzer | js | output_encoding | GHSA-pgr8-jg6h-8gw6 | `src/viewer.js::chartData` |
| zenitram/malla | js | output_encoding | GHSA-ch57-39q2-4crm | `src/malla/static/js/chat.js::refreshPop` (+12) |
| zereight/gitlab-mcp | ts | path | GHSA-cv3r-c5h8-f4g5 | `index.ts::startSSEServer` (+1) |

"(+N)" means the fix touched N more functions the verifier also accepted. They collapse into one
positive change per fix commit in the manifest; the extra functions are listed as `targets` on that
change.

## Decisions made while building

- **Parent is `fix^1`, read from git**, not the `parent` field stored in `pool-p1`. That field did
  not always equal the fix commit's real first parent. "The code just before the fix" is `fix^1`.
- **Candidates are every function the fix touched.** `fix_sites()` does not know which function
  holds the vulnerability, so the catalog over-generates and the LLM check plus the human spot-check
  select the real location. One repo (`zenitram/malla`) produced 40 candidates from one fix.
- **The negative ratio is per positive change**, not per accepted function. `assemble.py` was
  corrected; the old count would have drawn 260 negatives for malla alone.
- **The held-out guard runs against training sources only.** `build_used()` also scans
  `measurements/` and `populations/`, which contain the SZZ pool and analysis of it, so it flagged all
  12 repos as "used" by my own earlier measurements. That is circular. None of the 12 appear in any
  training source (pairs, pointers, harvest, units, sweep).
- **Catalog enumeration skips a site it cannot read** (fix renamed or removed the function) instead
  of aborting the whole run. Verification stays strict.
- **The hook has no LLM detection tier.** `push/assistance.py` only picks witnesses for defects the
  engine already admitted. R6 was amended to measure the hook as shipped and the hook plus the
  hunter arm; the second is the number the tool is held to.
- **Hunter findings are classified, never relabelled.** The classifier gets the finding's excerpt
  and the taxonomy list; the change's own label is never shown to it. The raw hunter family, title
  and rationale stay on the defect for audit.
- **Replay output kept every result twice** (top level and `attempt-*/`), 30 MB a record, and the
  host hit 1.1 GB free at record 162. Byte-identical `attempt-*/` copies of finished records were
  removed; a guard repeats that while the replay runs. Nothing in Docker was touched.

## Not done yet

- Exploit-verified rate: the proof arm has not run on any of the 26 flagged positive findings.
- Growing from 12 to the ~40 positives `score.py:required_n` asks for.
- The 252-change replay at the engine's ~200 s fixed cost is about 17 hours one container at a time
  on the local host; that scale belongs on the kube `ax` lane, still blocked on the gVisor
  checkpoint path.

## Reproduce

```
# clones: fetch each repo single-branch with history, then fetch <fix> <parent>
python -m benchmarks.valid.catalog  sources.json --repos repos.json --out catalog.toml
DEEPSEEK_API_KEY=... python -m benchmarks.valid.verify run catalog.toml --repos repos.json --out verification.json
python -m benchmarks.valid.verify spot-check verification.json <id> accept --reviewer mc --date 2026-10-12
python - <<'PY'   # assemble with the training-only used set
from benchmarks.valid import assemble
...  # see this directory's provenance files; used = build_used(root) minus {measurements, populations}
PY
# arm (a): replay.py --arm long_deadline, docker lane, --docker-image-bytes 3608729816, output outside the repo
# arm (b): DEEPSEEK_API_KEY=... python -m benchmarks.valid.hunter_arm run benchmarks/valid/pilot/pool-valid.toml --slice 1 --repos repos_hist.json --out <hunter_out>
python -m benchmarks.unseen.score <records_dir> --out scored.json --slice-id 1
python -m benchmarks.valid.metrics <records_dir> --scored scored.json --out metrics.json
```