mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5
1af3db37f6
output_encoding had exactly ONE pair in the whole corpus. It now has 41. 238 pairs were harvested from OWASP BenchmarkJava (2740 generated servlets, GPL-2.0, with a ground-truth CSV) by matching each really-vulnerable test to the safe test of the same category it most resembles -- adjacent tests come from one template, so they differ in the sink or the sanitiser rather than being unrelated programs. Median similarity 0.91, against the 64% of the older `sast` OWASP entries that the group-4 stratification flagged as different programs. Only 40 are vendored. One representative of each of the seven categories was run through the model BEFORE vendoring, and only xss distinguished: its fix adds ESAPI encoding, which the Java sanitizer facts recognise, so the vulnerable side entails and the fixed side drops to corroborated. cmdi, sqli, ldapi and xpathi put the sink in a helper class outside the servlet and resolved no flow at all; pathtraver found the flow but its "safe" twin carries no distinguishing call, so text similarity had matched a twin that is not semantically the fix -- a subtler form of the very defect the harvest was designed to avoid. Those 198 pairs are deliberately left out: adding pairs the model cannot read would inflate the corpus while making every number less interpretable. Evidence for the vendored slice is 3 of 3 distinguished, which is a small sample and is stated as such in the catalog header rather than implied to be more. The slice is marked SYNTHETIC in its header: it measures breadth across a weakness class and must never stand in for real-world performance. The corpus already shows a +16% overfitting gap, and a large generated slice blended into a headline would widen it invisibly. Two things recorded rather than hidden: BenchmarkJava's LICENSE is GPL-2.0 while the older sast entries declare GPL-3.0 for OWASP, which looks like an error and is not copied; and OWASP-Benchmark/BenchmarkJava already straddled the train/holdout split before these pairs (owasp-java-cmdi train, owasp-java-sqli holdout), so the new all-holdout pairs enlarge an existing straddle rather than create one. The two vendored-count guards did their job -- a corpus change could not land silently -- and are updated as the acknowledgement. Hygiene artifact regenerated: 277 -> 317 vendored pairs. 720 passing, mypy/ruff clean, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG