- Pair eval already concatenates `benchmarks/pairs/catalog.toml` with `sast/catalog.toml`. A third file `vfc/catalog.toml` is the smallest seam.
- CLI `--slice` choices are a closed set (`all`, `local`, `github`, `sast`). `vfc` must be added there and in tests that assert slice names.
-`sast` already scores overlay with inventory fallback when nothing is promoted or demoted. `vfc` should use that path as the primary scorer and add a second inventory-only figure for the honesty dashboard.
-`pair_gate` evaluates the full catalog but fails CI only on `local` 100%. Adding vfc pairs does not change the fail condition if names stay out of `LANGUAGE_MANIFESTS`.
- Seed commits with public VCS snapshots exist: OpenSSL Heartbleed (`tls1_process_heartbeat`), Firefox libmar (`mar_insert_item`, CVE-2020-15667), Chromium FileReader (`ArrayBufferResult`, CVE-2019-5786). All three keep a sink-shaped call after the fix, so overlay/inventory may LEAK; that is the dashboard, not a 100% gate.
- Heartbleed fix adds a record-length check; `memcpy(bp, pl, payload)` remains. Inventory `c-unsafe-memory-copy` will fire on both sides (LEAK) until CWE-121 semantics exist.
- Firefox fix is `int namelen` → `uint32_t namelen`; `memcpy(item->name, name, namelen + 1)` remains. Same LEAK class.
- Chromium fix copies the ArrayBuffer on partial results (UAF). No memcpy/eval sink; expected MISS on inventory and overlay. Still a labeled pair.
- GitHub raw and Phabricator `?diff=1` return the function text without cloning.
- Implications: Seed three reviewed pairs. Tests assert presence and that the slice **runs**, not that pair_correct is 100%. Memory-safety pairs are allowed as known leaks (Req 6.3).
| Concatenate a `vfc` catalog like `sast` | Third TOML file, same `[[pair]]` schema | Matches existing loader; no new types | CLI choices and tests must be updated | **Selected** |
| Stuff vfc rows into `catalog.toml` | One file | Fewer paths | Mixes local/github with large-project VFCs | Rejected; sast already split |
| New eval package | Separate scorer | Clean isolation | Duplicates pair eval | Rejected |
| Clone megarepos in CI | Scan real trees | Maximum fidelity | Multi-GB, scrubbed commit messages, not isolated | Explicitly out of scope |
- Context: Need a labeled corpus that is offline, isolated, and honest about overlay versus regex.
- Alternatives Considered:
1. Inventory-only like `github`: misses the overlay question the user is evolving.
2. Overlay-only like `sast` without a sidecar: cannot see whether overlay helped or hurt.
3. Dual scoreboard with overlay primary and inventory sidecar: **selected**.
- Selected Approach: `slice = "vfc"` uses the overlay pair path (inventory fallback when unadjudicated). `PairEvalResult.scorers["vfc"]` holds both `overlay` and `inventory` metrics. Pair gate still fails only on `local`.
- Rationale: Reuses `sast` overlay machinery; satisfies Req 4 without making Youden a merge gate.
- Trade-offs: Pair gate runtime grows by three small overlay scans. Acceptable.
- Follow-up: Do not retune memcpy rules to force these pairs to PASS.
- Context: Grow the catalog later without cloning mozilla-central.
- Selected Approach: `benchmarks/pairs/vfc/harvest.py` plus `recipes.toml`. CI never runs harvest. Seed files are committed excerpts with provenance headers.
- Rationale: Req 5 and Req 7 (offline testability).
- Trade-offs: Harvest needs network when a maintainer grows the set; tests mock extraction locally.
-`pair_gate` currently scans the full catalog; three extra pairs are small. If the catalog grows, gate should keep evaluating local-only for the fail condition (already true) and may later skip overlay slices for speed.
