output_encoding had exactly ONE pair in the whole corpus. It now has 41.
238 pairs were harvested from OWASP BenchmarkJava (2740 generated servlets,
GPL-2.0, with a ground-truth CSV) by matching each really-vulnerable test to
the safe test of the same category it most resembles -- adjacent tests come
from one template, so they differ in the sink or the sanitiser rather than
being unrelated programs. Median similarity 0.91, against the 64% of the older
`sast` OWASP entries that the group-4 stratification flagged as different
programs.
Only 40 are vendored. One representative of each of the seven categories was
run through the model BEFORE vendoring, and only xss distinguished: its fix
adds ESAPI encoding, which the Java sanitizer facts recognise, so the
vulnerable side entails and the fixed side drops to corroborated. cmdi, sqli,
ldapi and xpathi put the sink in a helper class outside the servlet and
resolved no flow at all; pathtraver found the flow but its "safe" twin carries
no distinguishing call, so text similarity had matched a twin that is not
semantically the fix -- a subtler form of the very defect the harvest was
designed to avoid. Those 198 pairs are deliberately left out: adding pairs the
model cannot read would inflate the corpus while making every number less
interpretable.
Evidence for the vendored slice is 3 of 3 distinguished, which is a small
sample and is stated as such in the catalog header rather than implied to be
more.
The slice is marked SYNTHETIC in its header: it measures breadth across a
weakness class and must never stand in for real-world performance. The corpus
already shows a +16% overfitting gap, and a large generated slice blended into
a headline would widen it invisibly.
Two things recorded rather than hidden: BenchmarkJava's LICENSE is GPL-2.0
while the older sast entries declare GPL-3.0 for OWASP, which looks like an
error and is not copied; and OWASP-Benchmark/BenchmarkJava already straddled
the train/holdout split before these pairs (owasp-java-cmdi train,
owasp-java-sqli holdout), so the new all-holdout pairs enlarge an existing
straddle rather than create one.
The two vendored-count guards did their job -- a corpus change could not land
silently -- and are updated as the acknowledgement. Hygiene artifact
regenerated: 277 -> 317 vendored pairs.
720 passing, mypy/ruff clean, gates byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
