bench(train): class-targeted sourcing, --family, from the public OSV dumps

sources --family <family> keeps only advisories whose CWE the taxonomy maps to that family, writes
sources_<family>.json beside the training set, and excludes the pilot, every repository the
benchmarks already name, the strict used set and every earlier batch; --prefilter-only runs the
dump pass without GitHub. Implemented by Codex (gpt-6-astra), gated here: names normalize rejects
exclude nothing instead of aborting, and the strict used set is in the exclusions.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 21:46:45 +02:00
co-authored by Claude Fable 5.1
parent 499f6c4474
commit 3d08d692ff
3 changed files with 340 additions and 38 deletions
+32
View File
@@ -192,3 +192,35 @@ python $B candidates benchmarks/valid/train/verification.json benchmarks/valid/t
The verifier ran before the overlap check, so 137 of the 553 calls went to locations that were dropped later
(134 by the overlap check, 3 by the merge). A fresh run should call `overlap` before `clone`.
## Grow one family
To source another access-control batch from the existing ecosystem dumps:
```sh
.venv/bin/python -m benchmarks.valid.train.sourcing sources \
--family access_control --osv-dir /path/to/dumps --target 120
```
The directory must contain `PyPI.zip`, `npm.zip`, `Maven.zip` and `Packagist.zip`. The family comes from
`taxonomy.family_of_cwe`; the command prints its CWE ids at startup. Family runs use only OSV records,
keep the existing single-family rule, and exclude the strict used set, the pilot's accepted repositories,
every repository named elsewhere in the development tree, and repositories in `sources.json` and
`sources_batch*.json` in both the work and training directories. These exclusions happen before GitHub checks.
The target defaults to 120 accepted advisories for a family run, or 75 without `--family`. Acceptance still
requires the existing GitHub repository, commit-parent and licence checks. Family output goes to
`benchmarks/valid/train/sources_access_control.json` and `sources_access_control_log.json`; the earlier
source files are unchanged. Repeating a family run replaces that family's output. `--family` and `--extend`
are mutually exclusive. Cloning, catalog building and verification remain separate maintainer steps.
For a network-free dump inspection, add `--prefilter-only`. It reads the same dumps and exclusions, prints
per-ecosystem record, rejection and prefilter counts, and writes an empty source list with `accepted: 0`
and `github_checks_run: 0` in the log. Prefilter passes are not accepted advisories. Missing exclusion files
or unreadable dumps fail the run rather than silently weakening the exclusions.
`overlap --family access_control <batch>` also writes only that family's source file; a batch can be an
absolute path to the family output. `merge --family access_control --catalogs ... --verifications ...`
uses that family's sources and writes `catalog_access_control.toml` and `verification_access_control.json`.
Both commands keep their drop logs in the work directory with the family in the filename. Use the same
`--family` throughout to preserve the original training batch.