Open_Ultra_SAST/benchmarks/valid/train/SOURCING.md

# Training set for the hunter learning loop: sourcing run (2026-10-13)

Spec task 4.1 of `.kiro/specs/hunter-learning-loop/` (requirements R2.1 to R2.3). This is the record of the
sourcing run. **The training set is not frozen yet.** The model checks are done; the maintainer's spot-check is
not, and `assemble` only freezes accepted locations. Even if every candidate is accepted, the set has **28
positive changes, short of the 40 R2.3 asks for.** A run on it is a dry run until more positives are sourced.

## Files

In the tree (`benchmarks/valid/train/`):

| File | What it is |
|---|---|
| `sourcing.py` | Every step of this run (`used`, `sources`, `clone`, `filter`, `overlap`, `merge`, `verify`, `candidates`, `precheck`) |
| `sources.json` | 163 advisories that survived every exclusion, in the pilot's `sources.json` shape plus `origin`, `ecosystem`, `license_spdx` |
| `catalog.toml` | 413 fix-site locations for those advisories (test, vendored and minified files removed) |
| `verification.json` | One verifier record per catalog location (all `rejected` or `pending_human`) |
| `CANDIDATES.md` | The 99 locations that passed all three model checks, by repository, for the spot-check |

Out of the tree (`~/ousast-results/valid/train/`): `hist/<owner>__<name>` clones, `repos_hist.json`, and
`work/`, which holds the exclusion lists (`used.json`, `used_strict.json`, `named_elsewhere.json`), the per-batch
sources, catalogs and verifications, the drop lists, the clone log and the run logs. These stay out of the tree on
purpose: the repository guards in `tests/` scan `benchmarks/` for repository names, and an exclusion list is by
construction a list of names used elsewhere.

## Counts at every step

**Candidates.** Three batches, all with seed 1, in the order the task gave:

| Origin | Sourced | Dropped by the overlap check | Kept |
|---|---|---|---|
| `pool-fresh` (permissive `pool-p1` entries the pilot never sourced) | 17 | 17 | 0 |
| `pool-retry` (pilot-sourced, verifier rejected, not held out) | 20 | 20 | 0 |
| `osv` (batch 1, OSV dump) | 75 | 11 | 64 |
| `osv-extend` (batches 2 and 3, same order continued) | 100 | 1 | 99 |
| **Total** | **212** | **49** | **163** |

OSV source: the public bulk dumps (`osv-vulnerabilities.storage.googleapis.com/<ecosystem>/all.zip`, no token)
for PyPI, npm, Maven and Packagist. Go was left out because the catalog and the function cutter do not read Go
fix sites. The filter is `benchmarks/unseen/source.py:candidate()`, run unchanged: GitHub-reviewed, not
withdrawn, published 2019 or later, exactly one taxonomy family from the CWEs, exactly one GitHub fix-commit
reference, public repository under 500 MB, fix with exactly one parent, and a permissive licence read at the
fix's parent (unknown licences are skipped). One advisory per repository. Over the three batches the GitHub
checks rejected 306 advisories for a non-permissive or unknown licence, 71 for an ambiguous version bracket, 74
for a merge or root fix, 64 for the repository check, 2 not found, and 4 for transient `http_failure`. GitHub
calls: 584, 710 and 593 for the three batches (the token came from `gh auth token`). A first run of batch 1 read
only npm and Maven, because OSV calls the other two ecosystems `PyPI` and `Packagist` while the shared code
expects GitHub's `pip` and `composer`. The adapter maps the names now, and batch 1 was re-run before anything
was cloned.

Kept, by ecosystem: npm 52, Maven 49, pip 41, Composer 21.

**Overlap check (pilot and corpus exclusion).** A candidate is dropped if anything in the development tree
already names its repository. That means `github.com/<name>` or a `repo = "<name>"` line anywhere under
`benchmarks/`, `src/`, `tests/`, `plane/` or `.kiro/` (vendored trees skipped, as the pair guard skips them),
or a name in the strict used set (`build_used(root)` with every label). This removed:

- all 17 `pool-fresh` and all 20 `pool-retry`: they are reserved `pool-p1` repositories, and the retries are
  also named by the pilot's measurement files;
- 11 `osv`: hkuds/lightrag, jenkinsci/job-config-history-plugin, decolua/9router, code16/sharp,
  chainlit/chainlit, mineadmin/mineadmin, netflix/lemur, pheditor/pheditor, mockoon/mockoon and
  xorbitsai/inference (named in `benchmarks/independent` or `benchmarks/unseen`), plus startreedata/mcp-pinot,
  whose advisory text links jlowin/fastmcp, a repository in the `advisory-fixes-2` and `-3` pair corpora;
- 1 `osv-extend`: googleapis/google-oauth-java-client (named in `benchmarks/unseen`).

The first `used` step built a looser set that left out free text under `benchmarks/measurements` and
`benchmarks/independent`. Pool and population repositories got through that set into the first sourcing pass,
and the overlap check removed them afterwards. The merge step also dropped 3 catalog locations whose code
comments link a repository named elsewhere: jenkinsci/pipeline-maven-plugin (links jenkinsci/jenkins, and this
was its only passing location), snapappointments/bootstrap-select (links angular/angular) and
livehelperchat/livehelperchat (links axios/axios); the last two had been rejected anyway.

**Pilot exclusion check.** None of the 12 held-out pilot repositories appears in `sources.json`, `catalog.toml`,
`verification.json` or `CANDIDATES.md`: the overlap is 0, and `sourcing.py` asserts it. None of the 163 kept
repositories appears in the strict used set either.

**Clones.** 212 of 212 cloned with `--single-branch --no-tags` and full first-parent history. Each fix was
present or fetched, and the parent was read from git as `fix^1`; 0 of 212 differed from the stored parent. The
disk guard fired once: free space reached 6.0 GB after 64 clones, so cloning stopped. I then emptied the working
trees with `git sparse-checkout set --no-cone /__worktree_emptied_for_disk__`, which freed 3.4 GB. Nothing
downstream reads a checkout; `git sparse-checkout disable` restores one. Cloning then resumed with
`--no-checkout`. Free space later fell to 4.8 GB while the engine replay was writing, so clones of repositories
that could no longer contribute a positive were deleted (no catalog site, then no passing site, then not a
candidate after the overlap check). What is left: 29 clones, 2.4 GB, of which 28 are candidate repositories plus
pipeline-maven-plugin, whose only passing site was dropped. Peak use was about 6.5 GB. Host free space at the
end: 8.1 GB.

**Catalog.** Sites before filtering, then after:

| Batch | Sources | Raw sites | Test / vendored / minified dropped | Kept | Sources with a site |
|---|---|---|---|---|---|
| 1 | 112 | 463 | 112 / 5 / 1 | 345 | 75 |
| 2 | 60 | 217 | 74 / 0 / 2 | 141 | 34 |
| 3 | 40 | 111 | 38 / 3 / 3 | 67 | 30 |

Three sources hung in the catalog's regex function parser on large PHP files and were skipped after a 90 to 280 s
cap: tribalsystems/zenario, getgrav/grav-plugin-admin and genixcms/genixcms. After the overlap check and the
merge, the tree catalog holds **413 locations across 96 advisories and repositories**: Java 114, Python 104,
TypeScript 94, JavaScript 54, PHP 47.

`benchmarks/valid/catalog.py` now maps `.java` and `.php`. Before, it only read Python, JavaScript and
TypeScript, although `extract.fix_sites` and `function_bounds` read all five. The scope test in
`tests/test_valid_benchmark.py` now expects `.php` to give `php` and checks `.rb` as the out-of-scope suffix.

**Verification.** `deepseek-flash` through the shipped `verify_case`, temperature 0, 1200-token output cap.

- Calls: **553 of the 600 budget** (345 + 141 + 67), with 0 provider errors and no exception types seen. Every
  call logs the function bytes read on both sides first, so no verdict comes from an unread function.
- Locations ordered round-robin by advisory, so one advisory with many sites could not use up the budget.
- On the 413 locations in the tree: 306 vulnerable before the fix, 259 where the fix addresses it (258 both),
  107 reachable from an in-repo entry point, **99 passing all three with an entry point**.
- Across all 553 calls (including dropped candidates): 433, 371 and 143 per check; 132 passing.

**Candidates for the spot-check (`CANDIDATES.md`).** 99 locations, 28 repositories, 28 positive changes (one
fix commit each).

- Repositories by language: Java 10, Python 9, TypeScript 7, JavaScript 2, PHP 1 (apache/atlas has both
  JavaScript and TypeScript sites).
- Families: access_control 7, path 6, output_encoding 4, injection 4, config_secrets 4,
  untrusted_destination 3.
- Origin: 13 from batch 1 (`osv`), 15 from batches 2 and 3 (`osv-extend`).

## What blocks the freeze

1. **Human acceptance.** `assemble.freeze` only takes `accepted` verdicts, so I stopped before assemble. No
   spot-check was run.
2. **Too few positives.** At most 28 changes, against the 40 R2.3 asks for. The yield was about 18% of sourced
   advisories (29 of 163 survivors had a passing site before the last drop). Reaching 40 accepted needs roughly
   70 to 100 more advisories, about 200 to 250 more verifier calls, and about 3 GB of transient clone space.
   With this run's 47 remaining calls, that is a new budget decision for the maintainer.
3. **Ordinary-push availability.** A pre-check of `extract.ordinary_history`, the function `assemble` uses, on
   the first two repositories alphabetically found two failures. Both stop the whole freeze (`freeze` raises):
   - andrea9293/mcp-documentation-server has 8 usable ordinary commits, short of the 20 needed;
   - apache/atlas fails with `Rejected: function_parse` after 456 s (a Python file in its history does not parse).

   dtwang/line-desktop-mcp (56 first-parent commits) and modoboa/modoboa-dmarc (87) are also likely short. I
   stopped the pre-check there: it parses every changed file of every first-parent commit, and five candidates
   have 7,000 to 22,600 commits (trigger.dev, inventree, apache/pdfbox, home-assistant/frontend,
   htmlunit/htmlunit). That would be hours of CPU next to the running replay. Expect `assemble` to be slow on
   those repositories, or reject them in the spot-check.

## What the maintainer runs after accepting

Spot-check each location in `CANDIDATES.md`:

```
.venv/bin/python -m benchmarks.valid.verify spot-check benchmarks/valid/train/verification.json <id> accept|reject --reviewer mc --date YYYY-MM-DD
```

Then freeze with ratio 20 and seed 1. The used set must be the pre-run snapshot `work/used_strict.json` plus the
12 pilot repositories. A fresh `build_used(root)` would now sweep `benchmarks/valid/train/` itself and drop every
training repository.

```
PYTHONPATH=. .venv/bin/python - <<'PY'
import json, tomllib
from pathlib import Path
from benchmarks.valid import assemble
from benchmarks.valid.catalog import local_git
H = Path.home() / "ousast-results/valid/train"
T = Path("benchmarks/valid/train")
used = set(json.loads((H / "work/used_strict.json").read_text()))
used |= set(json.loads(Path("benchmarks/measurements/2026-10-12-valid-benchmark-pilot/accepted_repos.json").read_text()))
repos = {k: local_git(Path(v)) for k, v in json.loads((H / "repos_hist.json").read_text()).items()}
record = assemble.freeze(H / "work/freeze", cases=tomllib.loads((T / "catalog.toml").read_text())["case"],
                         verifications=json.loads((T / "verification.json").read_text())["cases"],
                         repos=repos, used=used, seed=1, ratio=20)
(T / "pool-train.toml").write_bytes((H / "work/freeze/pool-valid.toml").read_bytes())
(T / "freeze-train.json").write_bytes((H / "work/freeze/freeze-valid.json").read_bytes())
print(json.dumps({k: record[k] for k in ("freeze_digest", "dropped_count", "label_check_counts")}))
PY
```

`dropped_count` should be 0. A non-zero count means an accepted repository is in the used snapshot, which the
overlap check should have prevented.

## Reproduce

```
B=benchmarks/valid/train/sourcing.py; W=~/ousast-results/valid/train/work
python $B used
python $B sources --osv-dir <dir with PyPI/npm/Maven/Packagist all.zip> --target 75 --seed 1      # batch 1
python $B sources --osv-dir ... --target 60 --seed 1 --extend                                    # batch 2, then 40 for batch 3
python $B clone [--sources <batch file>]
python -m benchmarks.valid.catalog <batch sources> --repos $W/repos_hist.json --out <raw.toml>   # per source, with a time cap
python $B filter <raw.toml> --out <catalog.toml>
DEEPSEEK_API_KEY=... python $B verify <catalog.toml> --repos $W/repos_hist.json --out <verification.json> --budget <n>
python $B overlap sources_batch1.json sources_batch2.json sources_batch3.json
python $B merge --catalogs catalog_batch{1,2,3}.toml --verifications verification_batch{1,2,3}.json
python $B candidates benchmarks/valid/train/verification.json benchmarks/valid/train/catalog.toml --out benchmarks/valid/train/CANDIDATES.md
```

The verifier ran before the overlap check, so 137 of the 553 calls went to locations that were dropped later
(134 by the overlap check, 3 by the merge). A fresh run should call `overlap` before `clone`.

## Grow one family

To source another access-control batch from the existing ecosystem dumps:

```sh
.venv/bin/python -m benchmarks.valid.train.sourcing sources \
  --family access_control --osv-dir /path/to/dumps --target 120
```

The directory must contain `PyPI.zip`, `npm.zip`, `Maven.zip` and `Packagist.zip`. The family comes from
`taxonomy.family_of_cwe`; the command prints its CWE ids at startup. Family runs use only OSV records,
keep the existing single-family rule, and exclude the strict used set, the pilot's accepted repositories,
every repository named elsewhere in the development tree, and repositories in `sources.json` and
`sources_batch*.json` in both the work and training directories. These exclusions happen before GitHub checks.

The target defaults to 120 accepted advisories for a family run, or 75 without `--family`. Acceptance still
requires the existing GitHub repository, commit-parent and licence checks. Family output goes to
`benchmarks/valid/train/sources_access_control.json` and `sources_access_control_log.json`; the earlier
source files are unchanged. Repeating a family run replaces that family's output. `--family` and `--extend`
are mutually exclusive. Cloning, catalog building and verification remain separate maintainer steps.

For a network-free dump inspection, add `--prefilter-only`. It reads the same dumps and exclusions, prints
per-ecosystem record, rejection and prefilter counts, and writes an empty source list with `accepted: 0`
and `github_checks_run: 0` in the log. Prefilter passes are not accepted advisories. Missing exclusion files
or unreadable dumps fail the run rather than silently weakening the exclusions.

`overlap --family access_control <batch>` also writes only that family's source file; a batch can be an
absolute path to the family output. `merge --family access_control --catalogs ... --verifications ...`
uses that family's sources and writes `catalog_access_control.toml` and `verification_access_control.json`.
Both commands keep their drop logs in the work directory with the family in the filename. Use the same
`--family` throughout to preserve the original training batch.