mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
Marius CiepluchandClaude Opus 5
8c387306be
The pair corpus scores excerpts: a vulnerable function and its fixed twin, handed to the engine with the answer already narrowed down. Nothing in it exercises a repository -- region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on fifteen lines still holds when the source, the sink and the guard live in three modules. benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char commit SHA, licence, and every known vulnerability with its file, function and line. `ousast repos` lists them; `--fetch` materialises one; CI does neither, and conftest clears the switch and redirects the cache so a developer with warm checkouts cannot read them by accident. Three rules the loader enforces rather than documents: - A branch or tag is rejected. Either would make the same recipe mean different code next month while the baseline it produced kept the old numbers. - `in_scope` must be stated per vulnerability, never defaulted. It decides whether "not found" is a miss or a non-question, which is the most important thing about the result. - A recipe that declares `detection` must name something in scope to detect. `ousast repos` does not trust a recipe either: against a materialised checkout it checks HEAD against the pin, that the licence file is there, and that each declared function appears above its declared line. That earned its place immediately -- the first libpng recipe named LICENSE.md, which is the filename at HEAD today and not at the pinned commit, where it is still LICENSE. Two checkouts, verified against the real repositories: - vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are already in the pair corpus, including the update_password miss diagnosed earlier, so a whole-repository run says directly whether a verdict survives the move off the excerpt. Its routes register through an OpenAPI operationId, so it exercises registration-based entry-point mapping rather than the decorator path. - libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at the parent of the commit that added the missing clamp. Its CVE is marked out of scope: an out-of-bounds read is not a family this engine arbitrates, and recording "not found" against it as a miss is the easiest way to make a measurement lie. Nothing is vendored; the recipe records the licence so an artifact can say what it ran against. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
48 lines
2.0 KiB
TOML
48 lines
2.0 KiB
TOML
# VAmPI -- a deliberately vulnerable Flask API, small enough to iterate on.
|
|
#
|
|
# This is the DETECTION checkout. The corpus already scores excerpts cut out of VAmPI, and one of them
|
|
# (users/update_password) was diagnosed as a miss. Scanning the whole repository asks the question the
|
|
# excerpts cannot: does a verdict survive when the route registration, the handler and the query live in
|
|
# three different modules, and when the region ranker has to find the handler rather than being handed it?
|
|
#
|
|
# Its routes are registered through an OpenAPI `operationId`, not a decorator, so it also exercises the
|
|
# registration-based entry-point mapping rather than the decorator path.
|
|
|
|
name = "vampi"
|
|
url = "https://github.com/erev0s/VAmPI.git"
|
|
commit = "f16052dce83f05847133ec98f01c5193a41de7d8"
|
|
language = "python"
|
|
license = "MIT"
|
|
license_file = "LICENSE"
|
|
measures = ["detection"]
|
|
note = """
|
|
520 lines across 11 Python files: too small to say anything about the envelope, and chosen for exactly that
|
|
reason -- a detection result here is not confounded by budget exhaustion or region ranking.
|
|
"""
|
|
|
|
# VAmPI serves the vulnerable code paths behind a global switch, `int(os.getenv('vulnerable', 1))` in app.py.
|
|
# The default is already vulnerable; it is pinned here because a measurement that silently ran the patched
|
|
# branch would report a clean repository and look like a detection failure.
|
|
[environment]
|
|
vulnerable = "1"
|
|
|
|
[[known]]
|
|
id = "VAmPI-SQLI"
|
|
family = "injection"
|
|
weakness = "SQL injection: the username is interpolated into the query string at line 72 and executed at 73"
|
|
file = "models/user_model.py"
|
|
function = "get_user"
|
|
line = 73
|
|
reference = "README.md 'List of Vulnerabilities'; the vulnerable branch is guarded by `if vuln:`"
|
|
in_scope = true
|
|
|
|
[[known]]
|
|
id = "VAmPI-BOLA"
|
|
family = "access_control"
|
|
weakness = "unauthorized password change: the path username is used instead of the authenticated token subject"
|
|
file = "api_views/users.py"
|
|
function = "update_password"
|
|
line = 187
|
|
reference = "README.md 'Unauthorized Password Change'; the non-vulnerable branch uses resp['sub']"
|
|
in_scope = true
|