marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(learning-harness): absence rows travel with their registration

Browse Source 
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.

The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.

Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 12:39:55 +02:00
co-authored by Claude Opus 5
parent 7a33756a7e
commit c2d9c4ad01
48 changed files with 1613 additions and 11776 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/learning-harness/design.md
+1
View File
@@ -158,6 +158,7 @@ tests/test_learning_families.py, test_learning_classify.py, test_learning_scorin
- `src/openultrasast/cli.py` — `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
- `pyproject.toml` — `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml` — the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
- **Cross-file registration context (amended during task 4.1).** Some frameworks keep the route table out of the handler module: connexion registers by `operationId` in an OpenAPI document, an Express application registers in a router file. A single-file excerpt of such a handler can never contain its registration, so Req 10.2 is unreachable for those rows without a second file. A recipe may therefore name `context = ["<repo-relative path>"]`; `benchmarks/pairs/harvest.py` fetches each document at the same two commits, extracts only the stanza that names the handler (`extract_registration_context`), and writes it under `<name>/context/<side>/<relpath>` with its own provenance header and the same redaction. The catalog carries `[[pair.context]]` rows, `PairCase.context_files` holds `(relpath, vuln, fixed)`, and `_materialize_side` copies them beside the excerpt at their repository-relative paths so the entry-point mapper sees the same layout the project has. `src/openultrasast/mapping.py` gains `_route_manifest_entry_points`, which reads `operationId` and the operation's `security` block out of a YAML or JSON document and attributes the record to the module the operation id points at. Trap pairs whose fixed side is a different, correctly guarded handler carry `fix_function` so the fixed side gets *its* registration rather than the vulnerable side's.
- `benchmarks/pairs/mechanisms.toml` — unchanged; families reference mechanism ids.
- `.kiro/steering/roadmap.md` — rewritten by `publish` from artifacts.
.kiro/specs/learning-harness/tasks.md
+13 -1
View File
@@ -137,7 +137,7 @@
- _Requirements: 11.1, 11.4_
- [ ] 4. Corpus repairs
- [ ] 4.1 Re-harvest absence rows with registration context
- [x] 4.1 Re-harvest absence rows with registration context
- Absence rows in vibe-py and agent-vfc gain their family and the handler-context mode in the recipes; the maintainer re-harvests them over the network; every re-harvested excerpt keeps its license line, passes redaction and keeps the decorators and registration statements; failed fetches keep the old excerpt with a recorded reason; catalogs regenerate idempotently.
- Observable: the vampi and threatbyte excerpts contain their registration lines, the entry-point mapper names their handlers, and the catalogs load with unchanged vendored counts.
- _Depends: 1.3_
@@ -207,3 +207,15 @@
- Task-plan updates from the review: 2.2 and 2.4 claim the two criteria the maintainer added (2.6 and 3.7); 3.6 claims Req 6.2 with an observable that every non-target family directory is byte-identical across a round; 3.7 claims Req 3.5's K floor, 9.6's pair identities and 11.3's trajectories.
- **Escalated to the maintainer**: Req 6.1 lists "a skills set" among a detector configuration's parts, and neither the approved design nor the code has one, although the repository ships a skill router. Either Req 6.1 drops the skills set or a task wires it; the implementation followed the approved design, so this is a spec-internal conflict rather than a deviation.
- Carry-forward: the nineteen straddling repositories still cross the split by declaration and cap every holdout number these tasks produce; `unknown` serves as both the classifier's abstention and the generalist detector, which deserves a sentence in the design before a round ever targets it.
- Task 4.1 (2026-09-06, RED first: 21 of 23 failing tests in `tests/test_pair_registration_context.py`; then green, 26 tests). Measured before the change: **8 of the 10 vendored access-control rows had no named handler at all**, so every one of them would have scored `missing_context` forever rather than as a detector miss. After the re-harvest 9 of 10 are named and the tenth states its reason. What the repair needed, in order of how much it cost:
- **Class registration.** A flask-restx handler is a method; `@ns.route('/profile')` sits on the class. `extract_handler_context` now carries the enclosing class declaration and its decorators, so the excerpt is both reachable and parsable — a bare indented `def get` is neither.
- **Multi-line decorators and commented-out guards.** `@ns.doc(...)` wraps over four lines, and stopping at a continuation dropped the `@ns.route(...)` above it — the registration itself. `#@token_required` above a handler is not noise; in the two ThreatByte rows it is the entire bug, and the evidence field says so. `_decorator_start` now walks both.
- **Cross-file registration (design amendment, recorded in `design.md`).** connexion registers VAmPI's handlers by `operationId` in `openapi_specs/openapi3.yml`; Overleaf registers `exportProject` in `router.mjs`. No single-file excerpt can contain that, so a recipe may name `context = [...]`, the harvester writes the stanza per side under `<name>/context/<side>/<relpath>`, the catalog carries `[[pair.context]]`, and `_materialize_side` lays the documents out beside the excerpt at their repository-relative paths. `mapping._route_manifest_entry_points` reads `operationId` plus the operation's `security` block; `_js_statements` joins a registration that spans lines, because reading `webRouter.post(` alone finds neither the handler nor the middleware that guards it.
- **Anchors.** Real-Vuln-Benchmark trap rows take their fixed side from a different, correctly guarded handler of the same snapshot, so `handler_context` honours `line`/`fix_line` and a new `fix_function` gives the fixed side *its* registration instead of the vulnerable side's.
- **`innermost_function_bounds`** for brace languages, used only by this mode: `function_bounds_at` returns the outermost top-level group, which for a handler declared inside a React component is the whole 6000-line component. `hunk` and `enclosing` rows keep the bounds they were harvested with.
- **Fetch-then-write.** `harvest_recipe` fetches every blob before the first write, and the driver takes `--report` so a failed fetch leaves the reviewed excerpt exactly as it was and says why. A half-written re-harvest is indistinguishable from a real upstream change.
- Bugs the re-harvest exposed and fixed on the way: `.mjs` and `.cjs` were not in `LANGUAGE_BY_EXTENSION`, so the one pair vendored as an ES module had **no file target at all** — no entry point, no finding, a permanent miss; and `_python_route_access` matched auth words anywhere in a decorator, so `@ns.doc(description='... without proper authorization checks')` read the unguarded ThreatByte endpoint as `authenticated`, silencing every obligation on it. The guard is now the decorator's name, never its prose, and `@token_required` — the commonest Flask guard — is recognised at all.
- Corpus corrections: `lensops-kubernetesmanagement-fe4910` was labeled `setErrorMessage`, a React state setter, because the hunk landed in the wrong bounds; it is now `loadNamespaces` and the pair diff is exactly the fix. Its bug is a browser-side call omitting the header its siblings carry, not a missing guard on an inbound handler, so it carries `known_limit = "client_side_caller"` and leaves the denominators with a stated reason.
- Three existing tests encoded the old corpus and now encode the new one: the TypeScript parse pin (the lensops excerpt parses now), the classifier report (`labeled` 0 -> 4, the four seeded rows; the six `title`-tier agent rows carry a family but are not labels a maintainer stands behind), and the hygiene report ordering.
- **New honesty finding, recorded in the hygiene artifact as `cross_split_duplicates`: nine of the ten duplicate groups sit on both sides of the split.** Two forks of one project (`openniw-jobs-0fc947` train, `openniw-jobs-b74de3` holdout) and eight CVEs harvested twice under different names are byte-identical, so the train row teaches the holdout row exactly. `split_by_repository` cannot see it — the duplicates are different repositories. `duplicate_groups` is now ordered by name so the committed report does not reshuffle when one excerpt is re-harvested.
- Pre-existing drift the regeneration corrected, unrelated to this task: `catalog.toml` in git disagreed with a fresh `catalog_gen` run at the group-3 tip on two derived labels (`origin` -> `createApplication`, `normalizeDomain` -> `getGoogleProvider`). Verified in a detached worktree at `7a33756` before touching anything.
benchmarks/measurements/2026-09-06-corpus-hygiene.json
+66 -28
View File
@@ -1,45 +1,83 @@
{
"command": "python -c 'from openultrasast.pairs import ...; duplicate_groups / split_by_repository over the vendored catalog'",
"duplicate_groups": [
[
"openssl-cve-2023-0286",
"openssl-cve-2023-0286-general_name_cmp"
],
[
"openniw-jobs-0fc947",
"openniw-jobs-b74de3"
],
"cross_split_duplicates": [
[
"curl-cve-2023-38545",
"curl-cve-2023-38545-do_socks5"
],
[
"openssl-cve-2021-3712",
"openssl-cve-2021-3712-append_ia5"
],
[
"openssl-cve-2020-1967",
"openssl-cve-2020-1967-tls1_check_sig_alg"
],
[
"openssl-cve-2022-3786",
"openssl-cve-2022-3786-ossl_punycode_decode"
],
[
"openssl-cve-2020-1971",
"openssl-cve-2020-1971-general_name_cmp"
],
[
"etude-lexical-markdown-editorconfig-c61a8b",
"popmark-editorconfig-e5ac40"
],
[
"anastasios-pw-session-14ec1a",
"astroclaw-pw-session-1553a8"
"openniw-jobs-0fc947",
"openniw-jobs-b74de3"
],
[
"openssl-cve-2020-1967",
"openssl-cve-2020-1967-tls1_check_sig_alg"
],
[
"openssl-cve-2020-1971",
"openssl-cve-2020-1971-general_name_cmp"
],
[
"openssl-cve-2021-3712",
"openssl-cve-2021-3712-append_ia5"
],
[
"openssl-cve-2022-3602",
"openssl-cve-2022-3602-ossl_punycode_decode"
],
[
"openssl-cve-2022-3786",
"openssl-cve-2022-3786-ossl_punycode_decode"
],
[
"openssl-cve-2023-0286",
"openssl-cve-2023-0286-general_name_cmp"
]
],
"duplicate_groups": [
[
"anastasios-pw-session-14ec1a",
"astroclaw-pw-session-1553a8"
],
[
"curl-cve-2023-38545",
"curl-cve-2023-38545-do_socks5"
],
[
"etude-lexical-markdown-editorconfig-c61a8b",
"popmark-editorconfig-e5ac40"
],
[
"openniw-jobs-0fc947",
"openniw-jobs-b74de3"
],
[
"openssl-cve-2020-1967",
"openssl-cve-2020-1967-tls1_check_sig_alg"
],
[
"openssl-cve-2020-1971",
"openssl-cve-2020-1971-general_name_cmp"
],
[
"openssl-cve-2021-3712",
"openssl-cve-2021-3712-append_ia5"
],
[
"openssl-cve-2022-3602",
"openssl-cve-2022-3602-ossl_punycode_decode"
],
[
"openssl-cve-2022-3786",
"openssl-cve-2022-3786-ossl_punycode_decode"
],
[
"openssl-cve-2023-0286",
"openssl-cve-2023-0286-general_name_cmp"
]
],
"generated": "2026-09-06",
@@ -53,7 +91,7 @@
"known_limit": [
"owasp-java-hash"
],
"note": "Repositories in `straddling` have pairs on both sides of the train/holdout split, so a learning path can be taught by one file of a repository and evaluated on another. Pairs in `duplicate_groups` share a vulnerable excerpt byte for byte after the provenance header is stripped. Both cap what any detector number can mean.",
"note": "Repositories in `straddling` have pairs on both sides of the train/holdout split, so a learning path can be taught by one file of a repository and evaluated on another. Pairs in `duplicate_groups` share a vulnerable excerpt byte for byte after the provenance header is stripped. Both cap what any detector number can mean. Groups in `cross_split_duplicates` sit on both sides of the split while being the same code, so the train row teaches the holdout row exactly; forks of one project defeat the repository-level split.",
"straddling": [
[
"Contrast-Security-OSS/DjanGoat",
benchmarks/pairs/agent-vfc/catalog.toml
+15 -4
View File
@@ -881,13 +881,15 @@ fix_policy = "silent"
provenance = "agent"
split = "holdout"
review_tier = "title"
known_limit = "client_side_caller"
[[pair.expected]]
cwe = "CWE-862"
class = "missing authorization"
path = "components/KubernetesManagement.tsx"
function = "setErrorMessage"
function = "loadNamespaces"
mechanism = "missing_auth_guard"
family = "access_control"
evidence = "Fix missing auth header on namespaces API call"
[[pair]]
@@ -1091,7 +1093,7 @@ review_tier = "title"
cwe = "CWE-16"
class = "permissive default"
path = "src/presentation/http/app.ts"
function = "origin"
function = "createApplication"
mechanism = "permissive_default"
evidence = "fix: allow patch requests through cors"
@@ -1214,6 +1216,7 @@ class = "insecure direct object reference"
path = "backend/app/routers/jobs.py"
function = "get_job"
mechanism = "identity_from_request_body"
family = "access_control"
evidence = "Fix IDOR on job endpoint: case-bound jobs require case ownership"
[[pair]]
@@ -1243,6 +1246,7 @@ class = "insecure direct object reference"
path = "backend/app/routers/jobs.py"
function = "get_job"
mechanism = "identity_from_request_body"
family = "access_control"
evidence = "Fix IDOR on job endpoint: case-bound jobs require case ownership"
[[pair]]
@@ -1266,12 +1270,18 @@ provenance = "agent"
split = "train"
review_tier = "title"
[[pair.context]]
relpath = "services/web/app/src/router.mjs"
vuln = "overleaf-with-claude-exportscontroller-3980b9/context/vuln/services/web/app/src/router.mjs"
fixed = "overleaf-with-claude-exportscontroller-3980b9/context/fixed/services/web/app/src/router.mjs"
[[pair.expected]]
cwe = "CWE-639"
class = "insecure direct object reference"
path = "services/web/app/src/Features/Exports/ExportsController.mjs"
function = "exportProject"
mechanism = "identity_from_request_body"
family = "access_control"
evidence = "Fix IDOR in exports by adding token verification (Issue #31637) (#32883)"
[[pair]]
@@ -1483,6 +1493,7 @@ class = "insecure direct object reference"
path = "app/api/customers/claims/route.ts"
function = "PATCH"
mechanism = "identity_from_request_body"
family = "access_control"
evidence = "Brief 62: Fix IDOR on customer_claims PATCH/DELETE (#156)"
[[pair]]
@@ -1512,7 +1523,7 @@ class = "insecure direct object reference"
path = "app/api/customers/opportunities/route.ts"
function = "GET"
mechanism = "identity_from_request_body"
sink = "path"
family = "access_control"
evidence = "Brief 60: Fix IDOR on opportunities GET/PATCH/DELETE (missing company membership checks) (#148)"
[[pair]]
@@ -1749,7 +1760,7 @@ review_tier = "title"
cwe = "CWE-89"
class = "sql injection"
path = "src/utils.ts"
function = "normalizeDomain"
function = "getGoogleProvider"
mechanism = "source_reaches_sink"
evidence = "Fix SQL injection in search by using parameterized queries (#41)"
benchmarks/pairs/agent-vfc/lensops-kubernetesmanagement-fe4910/fixed.tsx
+8 -5779
View File
File diff suppressed because it is too large Load Diff
benchmarks/pairs/agent-vfc/lensops-kubernetesmanagement-fe4910/vuln.tsx
+8 -5779
View File
File diff suppressed because it is too large Load Diff
benchmarks/pairs/agent-vfc/new-erp-final-app-6e9644/fixed.ts
+1 -1
View File
@@ -5,7 +5,7 @@
// commit_url: https://github.com/Sachin-Salgar/NEW_ERP_FINAL/commit/6e9644c81ce3fb356669157182a5ef09a70c4b59
// cve:
// license: Apache-2.0
// function: origin
// function: createApplication
// relpath: src/presentation/http/app.ts
// provenance: agent
// mechanism: permissive_default
benchmarks/pairs/agent-vfc/new-erp-final-app-6e9644/vuln.ts
+1 -1
View File
@@ -5,7 +5,7 @@
// commit_url: https://github.com/Sachin-Salgar/NEW_ERP_FINAL/commit/6e9644c81ce3fb356669157182a5ef09a70c4b59
// cve:
// license: Apache-2.0
// function: origin
// function: createApplication
// relpath: src/presentation/http/app.ts
// provenance: agent
// mechanism: permissive_default
benchmarks/pairs/agent-vfc/openniw-jobs-0fc947/fixed.py
+3 -1
View File
@@ -1,4 +1,4 @@
# Provenance: HHHHHejia/openniw (fixed).
# Provenance: HHHHHejia/openniw get_job (fixed).
# repo: HHHHHejia/openniw
# commit: 0fc94733f6ca3c6b129a0a8e6d7b3acce19d5a10
# parent: 4d91ed82fa9a5ec6bf5ef96da1d221fc187700dc
@@ -9,7 +9,9 @@
# relpath: backend/app/routers/jobs.py
# provenance: agent
# mechanism: identity_from_request_body
# upstream_start: 9
@router.get("/{job_id}")
async def get_job(job_id: str, user: dict = Depends(auth.current_user)) -> dict:
job = await jobs_service.get(job_id)
if job is None:
benchmarks/pairs/agent-vfc/openniw-jobs-0fc947/vuln.py
+3 -1
View File
@@ -1,4 +1,4 @@
# Provenance: HHHHHejia/openniw (vuln).
# Provenance: HHHHHejia/openniw get_job (vuln).
# repo: HHHHHejia/openniw
# commit: 4d91ed82fa9a5ec6bf5ef96da1d221fc187700dc
# parent: 4d91ed82fa9a5ec6bf5ef96da1d221fc187700dc
@@ -9,7 +9,9 @@
# relpath: backend/app/routers/jobs.py
# provenance: agent
# mechanism: identity_from_request_body
# upstream_start: 9
@router.get("/{job_id}")
async def get_job(job_id: str, user: dict = Depends(auth.current_user)) -> dict:
job = await jobs_service.get(job_id)
if job is None:
benchmarks/pairs/agent-vfc/openniw-jobs-b74de3/fixed.py
+3 -1
View File
@@ -1,4 +1,4 @@
# Provenance: ziyuzuo/openniw (fixed).
# Provenance: ziyuzuo/openniw get_job (fixed).
# repo: ziyuzuo/openniw
# commit: b74de31b0a5f37e4e929ba6aa74dbe0f5e6d9a68
# parent: 98a0c392ba96cd2c4aa746dd6fc2b183f385225e
@@ -9,7 +9,9 @@
# relpath: backend/app/routers/jobs.py
# provenance: agent
# mechanism: identity_from_request_body
# upstream_start: 9
@router.get("/{job_id}")
async def get_job(job_id: str, user: dict = Depends(auth.current_user)) -> dict:
job = await jobs_service.get(job_id)
if job is None:
benchmarks/pairs/agent-vfc/openniw-jobs-b74de3/vuln.py
+3 -1
View File
@@ -1,4 +1,4 @@
# Provenance: ziyuzuo/openniw (vuln).
# Provenance: ziyuzuo/openniw get_job (vuln).
# repo: ziyuzuo/openniw
# commit: 98a0c392ba96cd2c4aa746dd6fc2b183f385225e
# parent: 98a0c392ba96cd2c4aa746dd6fc2b183f385225e
@@ -9,7 +9,9 @@
# relpath: backend/app/routers/jobs.py
# provenance: agent
# mechanism: identity_from_request_body
# upstream_start: 9
@router.get("/{job_id}")
async def get_job(job_id: str, user: dict = Depends(auth.current_user)) -> dict:
job = await jobs_service.get(job_id)
if job is None:
benchmarks/pairs/agent-vfc/overleaf-with-claude-exportscontroller-3980b9/context/fixed/services/web/app/src/router.mjs
+17
View File
@@ -0,0 +1,17 @@
// Provenance: ZUENS2020/overleaf-with-claude exportProject (fixed).
// repo: ZUENS2020/overleaf-with-claude
// commit: 3980b9e5806defb63e45ae2696e6c1899f3a8431
// parent: 5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed
// commit_url: https://github.com/ZUENS2020/overleaf-with-claude/commit/3980b9e5806defb63e45ae2696e6c1899f3a8431
// cve:
// license: AGPL-3.0
// function: exportProject
// relpath: services/web/app/src/router.mjs
// provenance: agent
// mechanism: identity_from_request_body
webRouter.post(
'/project/:project_id/export/:brand_variation_id',
AuthorizationMiddleware.ensureUserCanWriteProjectContent,
ExportsController.exportProject
)
benchmarks/pairs/agent-vfc/overleaf-with-claude-exportscontroller-3980b9/context/vuln/services/web/app/src/router.mjs
+17
View File
@@ -0,0 +1,17 @@
// Provenance: ZUENS2020/overleaf-with-claude exportProject (vuln).
// repo: ZUENS2020/overleaf-with-claude
// commit: 5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed
// parent: 5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed
// commit_url: https://github.com/ZUENS2020/overleaf-with-claude/commit/3980b9e5806defb63e45ae2696e6c1899f3a8431
// cve:
// license: AGPL-3.0
// function: exportProject
// relpath: services/web/app/src/router.mjs
// provenance: agent
// mechanism: identity_from_request_body
webRouter.post(
'/project/:project_id/export/:brand_variation_id',
AuthorizationMiddleware.ensureUserCanWriteProjectContent,
ExportsController.exportProject
)
benchmarks/pairs/agent-vfc/overleaf-with-claude-exportscontroller-3980b9/fixed.mjs
+5 -2
View File
@@ -1,4 +1,4 @@
// Provenance: ZUENS2020/overleaf-with-claude (fixed).
// Provenance: ZUENS2020/overleaf-with-claude exportProject (fixed).
// repo: ZUENS2020/overleaf-with-claude
// commit: 3980b9e5806defb63e45ae2696e6c1899f3a8431
// parent: 5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed
@@ -9,8 +9,9 @@
// relpath: services/web/app/src/Features/Exports/ExportsController.mjs
// provenance: agent
// mechanism: identity_from_request_body
// upstream_start: 1
import { expressify } from '@overleaf/promise-utils'
from '@overleaf/promise-utils'
import SessionManager from '../Authentication/SessionManager.mjs'
import logger from '@overleaf/logger'
import OError from '@overleaf/o-error'
@@ -80,3 +81,5 @@ async function exportProject(req, res, next) {
throw err
}
}
exportProject: expressify(exportProject),
benchmarks/pairs/agent-vfc/overleaf-with-claude-exportscontroller-3980b9/vuln.mjs
+5 -2
View File
@@ -1,4 +1,4 @@
// Provenance: ZUENS2020/overleaf-with-claude (vuln).
// Provenance: ZUENS2020/overleaf-with-claude exportProject (vuln).
// repo: ZUENS2020/overleaf-with-claude
// commit: 5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed
// parent: 5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed
@@ -9,8 +9,9 @@
// relpath: services/web/app/src/Features/Exports/ExportsController.mjs
// provenance: agent
// mechanism: identity_from_request_body
// upstream_start: 1
import { expressify } from '@overleaf/promise-utils'
from '@overleaf/promise-utils'
import SessionManager from '../Authentication/SessionManager.mjs'
import logger from '@overleaf/logger'
import OError from '@overleaf/o-error'
@@ -78,3 +79,5 @@ async function exportProject(req, res, next) {
throw err
}
}
exportProject: expressify(exportProject),
benchmarks/pairs/agent-vfc/recipes.toml
+22 -6
View File
@@ -904,7 +904,7 @@ repo = "amitknigam84/lensops"
parent = "9b87ecc3a44a7fe2a5185cca1fe52441647d5748"
commit = "fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84"
path = "components/KubernetesManagement.tsx"
mode = "hunk"
mode = "handler_context"
language = "typescript"
relpath = "components/KubernetesManagement.tsx"
license = "MIT"
@@ -920,6 +920,11 @@ trailer = "Co-Authored-By: Claude Sonnet 4.6 <[email protected]>"
reviewer = "pending"
evidence = "Fix missing auth header on namespaces API call"
commit_url = "https://github.com/amitknigam84/lensops/commit/fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84"
family = "access_control"
function = "loadNamespaces"
line = 805
fix_line = 805
known_limit = "client_side_caller"
split = "holdout"
[[recipe]]
@@ -1211,7 +1216,7 @@ repo = "HHHHHejia/openniw"
parent = "4d91ed82fa9a5ec6bf5ef96da1d221fc187700dc"
commit = "0fc94733f6ca3c6b129a0a8e6d7b3acce19d5a10"
path = "backend/app/routers/jobs.py"
mode = "hunk"
mode = "handler_context"
language = "python"
relpath = "backend/app/routers/jobs.py"
license = "MIT"
@@ -1227,6 +1232,8 @@ trailer = "Co-Authored-By: Claude Fable 5 <[email protected]>"
reviewer = "pending"
evidence = "Fix IDOR on job endpoint: case-bound jobs require case ownership"
commit_url = "https://github.com/HHHHHejia/openniw/commit/0fc94733f6ca3c6b129a0a8e6d7b3acce19d5a10"
family = "access_control"
function = "get_job"
split = "train"
[[recipe]]
@@ -1236,7 +1243,7 @@ repo = "ziyuzuo/openniw"
parent = "98a0c392ba96cd2c4aa746dd6fc2b183f385225e"
commit = "b74de31b0a5f37e4e929ba6aa74dbe0f5e6d9a68"
path = "backend/app/routers/jobs.py"
mode = "hunk"
mode = "handler_context"
language = "python"
relpath = "backend/app/routers/jobs.py"
license = "MIT"
@@ -1252,6 +1259,8 @@ trailer = "Co-Authored-By: Claude Fable 5 <[email protected]>"
reviewer = "pending"
evidence = "Fix IDOR on job endpoint: case-bound jobs require case ownership"
commit_url = "https://github.com/ziyuzuo/openniw/commit/b74de31b0a5f37e4e929ba6aa74dbe0f5e6d9a68"
family = "access_control"
function = "get_job"
split = "holdout"
[[recipe]]
@@ -1261,7 +1270,7 @@ repo = "ZUENS2020/overleaf-with-claude"
parent = "5a886aa9fb9fe4e2203f3cb1d62c91f4be0525ed"
commit = "3980b9e5806defb63e45ae2696e6c1899f3a8431"
path = "services/web/app/src/Features/Exports/ExportsController.mjs"
mode = "hunk"
mode = "handler_context"
language = "javascript"
relpath = "services/web/app/src/Features/Exports/ExportsController.mjs"
license = "AGPL-3.0"
@@ -1277,6 +1286,9 @@ trailer = "Co-authored-by: briangough <[email protected]
reviewer = "pending"
evidence = "Fix IDOR in exports by adding token verification (Issue #31637) (#32883)"
commit_url = "https://github.com/ZUENS2020/overleaf-with-claude/commit/3980b9e5806defb63e45ae2696e6c1899f3a8431"
family = "access_control"
function = "exportProject"
context = ["services/web/app/src/router.mjs"]
split = "train"
[[recipe]]
@@ -1468,7 +1480,7 @@ repo = "bigfivefoods/supplieradvisor-mvp"
parent = "a7a9bc260d649ea2e582c07435bfd7d93100647b"
commit = "387bcf40e984a2ee1832ab7beabf0284c5b627d8"
path = "app/api/customers/claims/route.ts"
mode = "hunk"
mode = "handler_context"
language = "typescript"
relpath = "app/api/customers/claims/route.ts"
license = "MIT"
@@ -1484,6 +1496,8 @@ trailer = "Co-authored-by: bigfivefoods <[email protected]
reviewer = "pending"
evidence = "Brief 62: Fix IDOR on customer_claims PATCH/DELETE (#156)"
commit_url = "https://github.com/bigfivefoods/supplieradvisor-mvp/commit/387bcf40e984a2ee1832ab7beabf0284c5b627d8"
family = "access_control"
function = "PATCH"
split = "holdout"
[[recipe]]
@@ -1493,7 +1507,7 @@ repo = "bigfivefoods/supplieradvisor-mvp"
parent = "8558538471e4fb10856f8d9a17d2be5ab5c4fb9c"
commit = "a69e8f27c0f4d402573d16f0612b51155bbc46e1"
path = "app/api/customers/opportunities/route.ts"
mode = "hunk"
mode = "handler_context"
language = "typescript"
relpath = "app/api/customers/opportunities/route.ts"
license = "MIT"
@@ -1509,6 +1523,8 @@ trailer = "Co-authored-by: bigfivefoods <[email protected]
reviewer = "pending"
evidence = "Brief 60: Fix IDOR on opportunities GET/PATCH/DELETE (missing company membership checks) (#148)"
commit_url = "https://github.com/bigfivefoods/supplieradvisor-mvp/commit/a69e8f27c0f4d402573d16f0612b51155bbc46e1"
family = "access_control"
function = "GET"
split = "train"
[[recipe]]
benchmarks/pairs/agent-vfc/supplieradvisor-mvp-route-387bcf/fixed.ts
+2 -1
View File
@@ -1,4 +1,4 @@
// Provenance: bigfivefoods/supplieradvisor-mvp (fixed).
// Provenance: bigfivefoods/supplieradvisor-mvp PATCH (fixed).
// repo: bigfivefoods/supplieradvisor-mvp
// commit: 387bcf40e984a2ee1832ab7beabf0284c5b627d8
// parent: a7a9bc260d649ea2e582c07435bfd7d93100647b
@@ -9,6 +9,7 @@
// relpath: app/api/customers/claims/route.ts
// provenance: agent
// mechanism: identity_from_request_body
// upstream_start: 90
export async function PATCH(request: NextRequest) {
try {
benchmarks/pairs/agent-vfc/supplieradvisor-mvp-route-387bcf/vuln.ts
+2 -1
View File
@@ -1,4 +1,4 @@
// Provenance: bigfivefoods/supplieradvisor-mvp (vuln).
// Provenance: bigfivefoods/supplieradvisor-mvp PATCH (vuln).
// repo: bigfivefoods/supplieradvisor-mvp
// commit: a7a9bc260d649ea2e582c07435bfd7d93100647b
// parent: a7a9bc260d649ea2e582c07435bfd7d93100647b
@@ -9,6 +9,7 @@
// relpath: app/api/customers/claims/route.ts
// provenance: agent
// mechanism: identity_from_request_body
// upstream_start: 90
export async function PATCH(request: NextRequest) {
try {
benchmarks/pairs/agent-vfc/supplieradvisor-mvp-route-a69e8f/fixed.ts
+2 -1
View File
@@ -1,4 +1,4 @@
// Provenance: bigfivefoods/supplieradvisor-mvp (fixed).
// Provenance: bigfivefoods/supplieradvisor-mvp GET (fixed).
// repo: bigfivefoods/supplieradvisor-mvp
// commit: a69e8f27c0f4d402573d16f0612b51155bbc46e1
// parent: 8558538471e4fb10856f8d9a17d2be5ab5c4fb9c
@@ -9,6 +9,7 @@
// relpath: app/api/customers/opportunities/route.ts
// provenance: agent
// mechanism: identity_from_request_body
// upstream_start: 89
export async function GET(request: NextRequest) {
try {
benchmarks/pairs/agent-vfc/supplieradvisor-mvp-route-a69e8f/vuln.ts
+2 -1
View File
@@ -1,4 +1,4 @@
// Provenance: bigfivefoods/supplieradvisor-mvp (vuln).
// Provenance: bigfivefoods/supplieradvisor-mvp GET (vuln).
// repo: bigfivefoods/supplieradvisor-mvp
// commit: 8558538471e4fb10856f8d9a17d2be5ab5c4fb9c
// parent: 8558538471e4fb10856f8d9a17d2be5ab5c4fb9c
@@ -9,6 +9,7 @@
// relpath: app/api/customers/opportunities/route.ts
// provenance: agent
// mechanism: identity_from_request_body
// upstream_start: 89
export async function GET(request: NextRequest) {
try {
benchmarks/pairs/agent-vfc/training/manifest.jsonl
+6 -6
View File
@@ -12,8 +12,8 @@
{"class": "prototype pollution", "commit": "c61a8bf78fe20d6ae0eab2e4b8b02004508525ed", "commit_url": "https://github.com/dayflower/etude-lexical-markdown/commit/c61a8bf78fe20d6ae0eab2e4b8b02004508525ed", "cve": "", "cwe": "CWE-1321", "file": "etude-lexical-markdown-editorconfig-c61a8b/fixed.ts", "function": "deepMergeInto", "id": "etude-lexical-markdown-editorconfig-c61a8b:fixed", "label": "fixed", "license": "MIT", "mechanism": "prototype_pollution", "pair": "etude-lexical-markdown-editorconfig-c61a8b", "project": "etude-lexical-markdown", "provenance": "agent", "relpath": "src/config/editorConfig.ts", "repo": "dayflower/etude-lexical-markdown", "review_tier": "title", "split": "train", "year": 2026}
{"class": "prototype pollution", "commit": "1c865815d05b072f4973388c3ac66b6696d9efde", "commit_url": "https://github.com/dayflower/etude-lexical-markdown/commit/db4f54b1ab80ad596de6425e84f90aa781c49999", "cve": "", "cwe": "CWE-1321", "file": "etude-lexical-markdown-editorconfig-db4f54/vuln.ts", "function": "deepMergeInto", "id": "etude-lexical-markdown-editorconfig-db4f54:vuln", "label": "vuln", "license": "MIT", "mechanism": "prototype_pollution", "pair": "etude-lexical-markdown-editorconfig-db4f54", "project": "etude-lexical-markdown", "provenance": "agent", "relpath": "src/config/editorConfig.ts", "repo": "dayflower/etude-lexical-markdown", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "prototype pollution", "commit": "db4f54b1ab80ad596de6425e84f90aa781c49999", "commit_url": "https://github.com/dayflower/etude-lexical-markdown/commit/db4f54b1ab80ad596de6425e84f90aa781c49999", "cve": "", "cwe": "CWE-1321", "file": "etude-lexical-markdown-editorconfig-db4f54/fixed.ts", "function": "deepMergeInto", "id": "etude-lexical-markdown-editorconfig-db4f54:fixed", "label": "fixed", "license": "MIT", "mechanism": "prototype_pollution", "pair": "etude-lexical-markdown-editorconfig-db4f54", "project": "etude-lexical-markdown", "provenance": "agent", "relpath": "src/config/editorConfig.ts", "repo": "dayflower/etude-lexical-markdown", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "missing authorization", "commit": "9b87ecc3a44a7fe2a5185cca1fe52441647d5748", "commit_url": "https://github.com/amitknigam84/lensops/commit/fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84", "cve": "", "cwe": "CWE-862", "file": "lensops-kubernetesmanagement-fe4910/vuln.tsx", "function": "setErrorMessage", "id": "lensops-kubernetesmanagement-fe4910:vuln", "label": "vuln", "license": "MIT", "mechanism": "missing_auth_guard", "pair": "lensops-kubernetesmanagement-fe4910", "project": "lensops", "provenance": "agent", "relpath": "components/KubernetesManagement.tsx", "repo": "amitknigam84/lensops", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "missing authorization", "commit": "fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84", "commit_url": "https://github.com/amitknigam84/lensops/commit/fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84", "cve": "", "cwe": "CWE-862", "file": "lensops-kubernetesmanagement-fe4910/fixed.tsx", "function": "setErrorMessage", "id": "lensops-kubernetesmanagement-fe4910:fixed", "label": "fixed", "license": "MIT", "mechanism": "missing_auth_guard", "pair": "lensops-kubernetesmanagement-fe4910", "project": "lensops", "provenance": "agent", "relpath": "components/KubernetesManagement.tsx", "repo": "amitknigam84/lensops", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "missing authorization", "commit": "9b87ecc3a44a7fe2a5185cca1fe52441647d5748", "commit_url": "https://github.com/amitknigam84/lensops/commit/fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84", "cve": "", "cwe": "CWE-862", "file": "lensops-kubernetesmanagement-fe4910/vuln.tsx", "function": "loadNamespaces", "id": "lensops-kubernetesmanagement-fe4910:vuln", "label": "vuln", "license": "MIT", "mechanism": "missing_auth_guard", "pair": "lensops-kubernetesmanagement-fe4910", "project": "lensops", "provenance": "agent", "relpath": "components/KubernetesManagement.tsx", "repo": "amitknigam84/lensops", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "missing authorization", "commit": "fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84", "commit_url": "https://github.com/amitknigam84/lensops/commit/fe49109aedfe21dea4d24d4d9f2fa3f1ecfb6c84", "cve": "", "cwe": "CWE-862", "file": "lensops-kubernetesmanagement-fe4910/fixed.tsx", "function": "loadNamespaces", "id": "lensops-kubernetesmanagement-fe4910:fixed", "label": "fixed", "license": "MIT", "mechanism": "missing_auth_guard", "pair": "lensops-kubernetesmanagement-fe4910", "project": "lensops", "provenance": "agent", "relpath": "components/KubernetesManagement.tsx", "repo": "amitknigam84/lensops", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "open redirect", "commit": "fec6f024ce72c9c47b61cbac6ce7a35cb2280602", "commit_url": "https://github.com/navaneethbv/market-cap/commit/4f8ae9be5519ced5a975f1190e3e4bc43f84e4d6", "cve": "", "cwe": "CWE-601", "file": "market-cap-route-4f8ae9/vuln.ts", "function": "GET", "id": "market-cap-route-4f8ae9:vuln", "label": "vuln", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "market-cap-route-4f8ae9", "project": "market-cap", "provenance": "agent", "relpath": "app/auth/confirm/route.ts", "repo": "navaneethbv/market-cap", "review_tier": "title", "split": "train", "year": 2026}
{"class": "open redirect", "commit": "4f8ae9be5519ced5a975f1190e3e4bc43f84e4d6", "commit_url": "https://github.com/navaneethbv/market-cap/commit/4f8ae9be5519ced5a975f1190e3e4bc43f84e4d6", "cve": "", "cwe": "CWE-601", "file": "market-cap-route-4f8ae9/fixed.ts", "function": "GET", "id": "market-cap-route-4f8ae9:fixed", "label": "fixed", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "market-cap-route-4f8ae9", "project": "market-cap", "provenance": "agent", "relpath": "app/auth/confirm/route.ts", "repo": "navaneethbv/market-cap", "review_tier": "title", "split": "train", "year": 2026}
{"class": "permissive default", "commit": "c074fc7413338aefcbea3170ef4d3ba486a8a4b7", "commit_url": "https://github.com/alexshao3/melodix/commit/089931ef71ea1e5b7586520baf1a5498e2f05b13", "cve": "", "cwe": "CWE-16", "file": "melodix-playerprovider-089931/vuln.tsx", "function": "PlayerProvider", "id": "melodix-playerprovider-089931:vuln", "label": "vuln", "license": "MIT", "mechanism": "permissive_default", "pair": "melodix-playerprovider-089931", "project": "melodix", "provenance": "agent", "relpath": "apps/miniapp/src/components/PlayerProvider.tsx", "repo": "alexshao3/melodix", "review_tier": "title", "split": "holdout", "year": 2026}
@@ -24,8 +24,8 @@
{"class": "prototype pollution", "commit": "4d4a5b063278c8b6697c3faae133d7c39fcd01c0", "commit_url": "https://github.com/andrebubniak/money-track/commit/4d4a5b063278c8b6697c3faae133d7c39fcd01c0", "cve": "", "cwe": "CWE-1321", "file": "money-track-category-icons-4d4a5b/fixed.ts", "function": "isCategoryIcon", "id": "money-track-category-icons-4d4a5b:fixed", "label": "fixed", "license": "MIT", "mechanism": "prototype_pollution", "pair": "money-track-category-icons-4d4a5b", "project": "money-track", "provenance": "agent", "relpath": "src/lib/category-icons.ts", "repo": "andrebubniak/money-track", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "prototype pollution", "commit": "d899b40d3466f2ffbe9f01c4d49267412401f078", "commit_url": "https://github.com/DavideCarvalho/nestjs-filter/commit/aa0e600d05997ea3d77538ea1c2db427c439f2e8", "cve": "", "cwe": "CWE-1321", "file": "nestjs-filter-normalizer-aa0e60/vuln.ts", "function": "normalizeInput", "id": "nestjs-filter-normalizer-aa0e60:vuln", "label": "vuln", "license": "MIT", "mechanism": "prototype_pollution", "pair": "nestjs-filter-normalizer-aa0e60", "project": "nestjs-filter", "provenance": "agent", "relpath": "packages/core/src/input/normalizer.ts", "repo": "DavideCarvalho/nestjs-filter", "review_tier": "title", "split": "train", "year": 2026}
{"class": "prototype pollution", "commit": "aa0e600d05997ea3d77538ea1c2db427c439f2e8", "commit_url": "https://github.com/DavideCarvalho/nestjs-filter/commit/aa0e600d05997ea3d77538ea1c2db427c439f2e8", "cve": "", "cwe": "CWE-1321", "file": "nestjs-filter-normalizer-aa0e60/fixed.ts", "function": "normalizeInput", "id": "nestjs-filter-normalizer-aa0e60:fixed", "label": "fixed", "license": "MIT", "mechanism": "prototype_pollution", "pair": "nestjs-filter-normalizer-aa0e60", "project": "nestjs-filter", "provenance": "agent", "relpath": "packages/core/src/input/normalizer.ts", "repo": "DavideCarvalho/nestjs-filter", "review_tier": "title", "split": "train", "year": 2026}
{"class": "permissive default", "commit": "08976c5d7a07b0663669b875b4507446199f85d7", "commit_url": "https://github.com/Sachin-Salgar/NEW_ERP_FINAL/commit/6e9644c81ce3fb356669157182a5ef09a70c4b59", "cve": "", "cwe": "CWE-16", "file": "new-erp-final-app-6e9644/vuln.ts", "function": "origin", "id": "new-erp-final-app-6e9644:vuln", "label": "vuln", "license": "Apache-2.0", "mechanism": "permissive_default", "pair": "new-erp-final-app-6e9644", "project": "NEW_ERP_FINAL", "provenance": "agent", "relpath": "src/presentation/http/app.ts", "repo": "Sachin-Salgar/NEW_ERP_FINAL", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "permissive default", "commit": "6e9644c81ce3fb356669157182a5ef09a70c4b59", "commit_url": "https://github.com/Sachin-Salgar/NEW_ERP_FINAL/commit/6e9644c81ce3fb356669157182a5ef09a70c4b59", "cve": "", "cwe": "CWE-16", "file": "new-erp-final-app-6e9644/fixed.ts", "function": "origin", "id": "new-erp-final-app-6e9644:fixed", "label": "fixed", "license": "Apache-2.0", "mechanism": "permissive_default", "pair": "new-erp-final-app-6e9644", "project": "NEW_ERP_FINAL", "provenance": "agent", "relpath": "src/presentation/http/app.ts", "repo": "Sachin-Salgar/NEW_ERP_FINAL", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "permissive default", "commit": "08976c5d7a07b0663669b875b4507446199f85d7", "commit_url": "https://github.com/Sachin-Salgar/NEW_ERP_FINAL/commit/6e9644c81ce3fb356669157182a5ef09a70c4b59", "cve": "", "cwe": "CWE-16", "file": "new-erp-final-app-6e9644/vuln.ts", "function": "createApplication", "id": "new-erp-final-app-6e9644:vuln", "label": "vuln", "license": "Apache-2.0", "mechanism": "permissive_default", "pair": "new-erp-final-app-6e9644", "project": "NEW_ERP_FINAL", "provenance": "agent", "relpath": "src/presentation/http/app.ts", "repo": "Sachin-Salgar/NEW_ERP_FINAL", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "permissive default", "commit": "6e9644c81ce3fb356669157182a5ef09a70c4b59", "commit_url": "https://github.com/Sachin-Salgar/NEW_ERP_FINAL/commit/6e9644c81ce3fb356669157182a5ef09a70c4b59", "cve": "", "cwe": "CWE-16", "file": "new-erp-final-app-6e9644/fixed.ts", "function": "createApplication", "id": "new-erp-final-app-6e9644:fixed", "label": "fixed", "license": "Apache-2.0", "mechanism": "permissive_default", "pair": "new-erp-final-app-6e9644", "project": "NEW_ERP_FINAL", "provenance": "agent", "relpath": "src/presentation/http/app.ts", "repo": "Sachin-Salgar/NEW_ERP_FINAL", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "server-side request forgery", "commit": "0c37bfd9e5292b15b1d7af51f52793d9bdf1cef0", "commit_url": "https://github.com/Irnhakim/nimestream/commit/d17f535bf3012041b056ff3aa991ff6cfa198e43", "cve": "", "cwe": "CWE-918", "file": "nimestream-route-d17f53/vuln.js", "function": "GET", "id": "nimestream-route-d17f53:vuln", "label": "vuln", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "nimestream-route-d17f53", "project": "nimestream", "provenance": "agent", "relpath": "app/api/img/route.js", "repo": "Irnhakim/nimestream", "review_tier": "title", "split": "train", "year": 2026}
{"class": "server-side request forgery", "commit": "d17f535bf3012041b056ff3aa991ff6cfa198e43", "commit_url": "https://github.com/Irnhakim/nimestream/commit/d17f535bf3012041b056ff3aa991ff6cfa198e43", "cve": "", "cwe": "CWE-918", "file": "nimestream-route-d17f53/fixed.js", "function": "GET", "id": "nimestream-route-d17f53:fixed", "label": "fixed", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "nimestream-route-d17f53", "project": "nimestream", "provenance": "agent", "relpath": "app/api/img/route.js", "repo": "Irnhakim/nimestream", "review_tier": "title", "split": "train", "year": 2026}
{"class": "insecure direct object reference", "commit": "4d91ed82fa9a5ec6bf5ef96da1d221fc187700dc", "commit_url": "https://github.com/HHHHHejia/openniw/commit/0fc94733f6ca3c6b129a0a8e6d7b3acce19d5a10", "cve": "", "cwe": "CWE-639", "file": "openniw-jobs-0fc947/vuln.py", "function": "get_job", "id": "openniw-jobs-0fc947:vuln", "label": "vuln", "license": "MIT", "mechanism": "identity_from_request_body", "pair": "openniw-jobs-0fc947", "project": "openniw", "provenance": "agent", "relpath": "backend/app/routers/jobs.py", "repo": "HHHHHejia/openniw", "review_tier": "title", "split": "train", "year": 2026}
@@ -52,7 +52,7 @@
{"class": "path traversal", "commit": "c3a0787080489b30e2ed0abfac03b51e45da184b", "commit_url": "https://github.com/D-sorganization/UpstreamDrift/commit/c3a0787080489b30e2ed0abfac03b51e45da184b", "cve": "", "cwe": "CWE-22", "file": "upstreamdrift-local-server-c3a078/fixed.py", "function": "_find_logo_file", "id": "upstreamdrift-local-server-c3a078:fixed", "label": "fixed", "license": "MIT", "mechanism": "path_join_user_input", "pair": "upstreamdrift-local-server-c3a078", "project": "UpstreamDrift", "provenance": "agent", "relpath": "src/api/local_server.py", "repo": "D-sorganization/UpstreamDrift", "review_tier": "title", "split": "train", "year": 2026}
{"class": "server-side request forgery", "commit": "934b939f1cca5259eaf807db3319eb97b90d8d80", "commit_url": "https://github.com/Ahkeratmehilaiset/waggledance-swarm/commit/9a454843aa35332ff2eb3106023eee0c4b5e444b", "cve": "", "cwe": "CWE-918", "file": "waggledance-swarm-ssrf-host-guard-9a4548/vuln.py", "function": "normalize_allowed_hosts", "id": "waggledance-swarm-ssrf-host-guard-9a4548:vuln", "label": "vuln", "license": "Apache-2.0", "mechanism": "source_reaches_sink", "pair": "waggledance-swarm-ssrf-host-guard-9a4548", "project": "waggledance-swarm", "provenance": "agent", "relpath": "waggledance/core/v3_13_0/ssrf_host_guard.py", "repo": "Ahkeratmehilaiset/waggledance-swarm", "review_tier": "title", "split": "train", "year": 2026}
{"class": "server-side request forgery", "commit": "9a454843aa35332ff2eb3106023eee0c4b5e444b", "commit_url": "https://github.com/Ahkeratmehilaiset/waggledance-swarm/commit/9a454843aa35332ff2eb3106023eee0c4b5e444b", "cve": "", "cwe": "CWE-918", "file": "waggledance-swarm-ssrf-host-guard-9a4548/fixed.py", "function": "normalize_allowed_hosts", "id": "waggledance-swarm-ssrf-host-guard-9a4548:fixed", "label": "fixed", "license": "Apache-2.0", "mechanism": "source_reaches_sink", "pair": "waggledance-swarm-ssrf-host-guard-9a4548", "project": "waggledance-swarm", "provenance": "agent", "relpath": "waggledance/core/v3_13_0/ssrf_host_guard.py", "repo": "Ahkeratmehilaiset/waggledance-swarm", "review_tier": "title", "split": "train", "year": 2026}
{"class": "sql injection", "commit": "eab923ad3099c01b002fbd6be7cf994614bbdee4", "commit_url": "https://github.com/G4brym/workers-research/commit/209444eb61cc564c2d6b43267256f85fe3e5e30e", "cve": "", "cwe": "CWE-89", "file": "workers-research-utils-209444/vuln.ts", "function": "normalizeDomain", "id": "workers-research-utils-209444:vuln", "label": "vuln", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "workers-research-utils-209444", "project": "workers-research", "provenance": "agent", "relpath": "src/utils.ts", "repo": "G4brym/workers-research", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "sql injection", "commit": "209444eb61cc564c2d6b43267256f85fe3e5e30e", "commit_url": "https://github.com/G4brym/workers-research/commit/209444eb61cc564c2d6b43267256f85fe3e5e30e", "cve": "", "cwe": "CWE-89", "file": "workers-research-utils-209444/fixed.ts", "function": "normalizeDomain", "id": "workers-research-utils-209444:fixed", "label": "fixed", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "workers-research-utils-209444", "project": "workers-research", "provenance": "agent", "relpath": "src/utils.ts", "repo": "G4brym/workers-research", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "sql injection", "commit": "eab923ad3099c01b002fbd6be7cf994614bbdee4", "commit_url": "https://github.com/G4brym/workers-research/commit/209444eb61cc564c2d6b43267256f85fe3e5e30e", "cve": "", "cwe": "CWE-89", "file": "workers-research-utils-209444/vuln.ts", "function": "getGoogleProvider", "id": "workers-research-utils-209444:vuln", "label": "vuln", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "workers-research-utils-209444", "project": "workers-research", "provenance": "agent", "relpath": "src/utils.ts", "repo": "G4brym/workers-research", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "sql injection", "commit": "209444eb61cc564c2d6b43267256f85fe3e5e30e", "commit_url": "https://github.com/G4brym/workers-research/commit/209444eb61cc564c2d6b43267256f85fe3e5e30e", "cve": "", "cwe": "CWE-89", "file": "workers-research-utils-209444/fixed.ts", "function": "getGoogleProvider", "id": "workers-research-utils-209444:fixed", "label": "fixed", "license": "MIT", "mechanism": "source_reaches_sink", "pair": "workers-research-utils-209444", "project": "workers-research", "provenance": "agent", "relpath": "src/utils.ts", "repo": "G4brym/workers-research", "review_tier": "title", "split": "holdout", "year": 2026}
{"class": "permissive default", "commit": "2604269350bff51f58572c133b45010cb0976a5a", "commit_url": "https://github.com/zoonk/zoonk/commit/bfbf670ccb41d63c407d32c51bd26e626a174f31", "cve": "", "cwe": "CWE-16", "file": "zoonk-origin-bfbf67/vuln.ts", "function": "getAllowedHosts", "id": "zoonk-origin-bfbf67:vuln", "label": "vuln", "license": "MIT", "mechanism": "permissive_default", "pair": "zoonk-origin-bfbf67", "project": "zoonk", "provenance": "agent", "relpath": "packages/utils/src/origin.ts", "repo": "zoonk/zoonk", "review_tier": "title", "split": "train", "year": 2026}
{"class": "permissive default", "commit": "bfbf670ccb41d63c407d32c51bd26e626a174f31", "commit_url": "https://github.com/zoonk/zoonk/commit/bfbf670ccb41d63c407d32c51bd26e626a174f31", "cve": "", "cwe": "CWE-16", "file": "zoonk-origin-bfbf67/fixed.ts", "function": "getAllowedHosts", "id": "zoonk-origin-bfbf67:fixed", "label": "fixed", "license": "MIT", "mechanism": "permissive_default", "pair": "zoonk-origin-bfbf67", "project": "zoonk", "provenance": "agent", "relpath": "packages/utils/src/origin.ts", "repo": "zoonk/zoonk", "review_tier": "title", "split": "train", "year": 2026}
benchmarks/pairs/agent-vfc/workers-research-utils-209444/fixed.ts
+1 -1
View File
@@ -5,7 +5,7 @@
// commit_url: https://github.com/G4brym/workers-research/commit/209444eb61cc564c2d6b43267256f85fe3e5e30e
// cve:
// license: MIT
// function: normalizeDomain
// function: getGoogleProvider
// relpath: src/utils.ts
// provenance: agent
// mechanism: source_reaches_sink
benchmarks/pairs/agent-vfc/workers-research-utils-209444/vuln.ts
+1 -1
View File
@@ -5,7 +5,7 @@
// commit_url: https://github.com/G4brym/workers-research/commit/209444eb61cc564c2d6b43267256f85fe3e5e30e
// cve:
// license: MIT
// function: normalizeDomain
// function: getGoogleProvider
// relpath: src/utils.ts
// provenance: agent
// mechanism: source_reaches_sink
benchmarks/pairs/catalog_gen.py
+18
View File
@@ -37,6 +37,14 @@ def excerpt_rel(recipe: dict[str, object]) -> tuple[str, str]:
return f"{name}/vuln{ext}", f"{name}/fixed{ext}"
def context_rel(recipe: dict[str, object]) -> list[tuple[str, str, str]]:
"""`(relpath, vuln document, fixed document)` for every document that registers the handler from outside its module."""
name = str(recipe["name"])
value = recipe.get("context") or ()
relpaths = (value,) if isinstance(value, str) else tuple(str(item) for item in value)
return [(relpath, f"{name}/context/vuln/{relpath}", f"{name}/context/fixed/{relpath}") for relpath in relpaths]
def _q(value: object) -> str:
return str(value).replace("\\", "\\\\").replace('"', '\\"')
@@ -85,6 +93,14 @@ def catalog_text(slice_name: str, recipes: list[dict[str, object]]) -> str:
lines.append(f'reviewer = "{_q(reviewer)}"')
if recipe.get("known_limit"):
lines.append(f'known_limit = "{_q(recipe["known_limit"])}"')
for relpath, context_vuln, context_fixed in context_rel(recipe):
lines += [
"",
"[[pair.context]]",
f'relpath = "{_q(relpath)}"',
f'vuln = "{_q(context_vuln)}"',
f'fixed = "{_q(context_fixed)}"',
]
lines += [
"",
"[[pair.expected]]",
@@ -95,6 +111,8 @@ def catalog_text(slice_name: str, recipes: list[dict[str, object]]) -> str:
if recipe.get("function"):
lines.append(f'function = "{_q(recipe["function"])}"')
lines.append(f'mechanism = "{_q(recipe.get("mechanism", "other"))}"')
if recipe.get("family"):
lines.append(f'family = "{_q(recipe["family"])}"') # Req 1: the family routes the work, never a CWE lookup
if recipe.get("rule_id"):
lines.append(f'rule_id = "{_q(recipe["rule_id"])}"')
if recipe.get("sink"):
benchmarks/pairs/harvest.py
+310 -28
View File
@@ -20,9 +20,11 @@ from __future__ import annotations
import argparse
import difflib
import json
import re
import sys
import tomllib
from collections.abc import Sequence
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
@@ -53,9 +55,16 @@ _SUFFIX_LANGUAGE = {
".py": "python",
".pl": "perl",
".pm": "perl",
".yml": "yaml",
".yaml": "yaml",
}
_HASH_COMMENT = {"python", "perl"}
_HASH_COMMENT = {"python", "perl", "yaml"}
_CONTEXT_LANGUAGES = {"yaml", "javascript", "typescript", "python"}
_DEF_RE = re.compile(r"^(\s*)(?:async\s+)?def\s+([A-Za-z_]\w*)\s*\(")
_CLASS_RE = re.compile(r"^(\s*)class\s+([A-Za-z_]\w*)\s*[(:]")
_COMMENTED_CODE = re.compile(r"^#\s*(?:@|(?:async\s+)?def\s|class\s|function\s)")
_CONTROL_HEAD = re.compile(r"^(?:else\s+)?(?:if|for|while|switch|catch|do|try|finally)\b")
_INDENTED = {"python", "yaml"}
class RecipeError(ValueError):
@@ -184,38 +193,230 @@ def extract_python_block(source: str, function_name: str) -> str:
if match is None or match.group(2) != function_name:
continue
indent = len(match.group(1))
start = index
while (
start > 0 and lines[start - 1].lstrip().startswith("@") and (len(lines[start - 1]) - len(lines[start - 1].lstrip())) == indent
):
start -= 1 # decorators belong to the function
start = _decorator_start(lines, index, indent)
end = _python_block_end(masked, index, indent)
return "".join(lines[start:end]).rstrip("\n") + "\n"
raise RecipeError(f"function not found: {function_name}")
def extract_handler_context(source: str, function_name: str, language: str = "c") -> str:
"""The named function (with its decorators) followed by every statement outside it that names the function as an
identifier: `app.add_url_rule(..., view_func=fn)`, `router.get('/x', fn)`, `path('x', fn)`, `app.use(fn)`. Comment and
string mentions never count (the masked text is searched). One excerpt per side, so guards living in decorators,
routers and middleware are visible to the teacher and the scorer (authorization-obligations, Req 7.1)."""
body = extract_function(source, function_name, language)
def _decorator_start(lines: Sequence[str], index: int, indent: int) -> int:
"""First line of the decorator run above ``index``: multi-line decorators and comments included.
A commented-out decorator is not noise: `#@token_required` above a handler is the whole bug in an absence row, and
an excerpt that starts below it hides the only trace the guard ever existed. `@ns.doc(...)` routinely wraps over
four lines, and stopping at its continuation drops the `@ns.route(...)` above it — the registration itself
(learning-harness Req 10.2)."""
first = index
cursor = index - 1
while cursor >= 0:
stripped = lines[cursor].strip()
if not stripped:
break
if _indent(lines[cursor]) == indent and stripped.startswith("@"):
first = cursor
cursor -= 1
continue
if _indent(lines[cursor]) == indent and stripped.startswith("#"):
if _COMMENTED_CODE.match(stripped):
first = cursor # `#@token_required` above the handler is the removed guard, not prose
cursor -= 1
continue
anchor = _decorator_anchor(lines, cursor, indent)
if anchor is None:
break
first = anchor
cursor = anchor - 1
return first
def _decorator_anchor(lines: Sequence[str], cursor: int, indent: int) -> int | None:
"""The `@` line of the multi-line decorator whose last line is ``cursor``, or None when ``cursor`` is not one."""
for start in range(cursor, -1, -1):
stripped = lines[start].strip()
if _indent(lines[start]) == indent and stripped.startswith("@"):
whole = " ".join(lines[position].strip() for position in range(start, cursor + 1))
opened = " ".join(lines[position].strip() for position in range(start, cursor))
balanced = whole.count("(") == whole.count(")") and opened.count("(") > opened.count(")")
return start if balanced else None
if stripped and _indent(lines[start]) <= indent:
return None
return None
def extract_handler_context(source: str, function_name: str, language: str = "c", *, line: int | None = None) -> str:
"""The named handler with everything that says it is reachable: its own decorators, the class its framework
registers (a `Resource`/`MethodView` method carries its route on the class, not on the function), and every
statement outside it that names the function as an identifier — `app.add_url_rule(..., view_func=fn)`,
`router.get('/x', fn)`, `path('x', fn)`, `app.use(fn)`. Comment and string mentions never count (the masked text
is searched). Pass ``line`` when the name repeats in the file: `def get` appears once per resource class, so the
name alone cannot say which handler a row labels. One excerpt per side, so guards living in decorators, routers
and middleware are visible to the teacher and the scorer (authorization-obligations Req 7.1, learning-harness
Req 10.2)."""
lines = source.splitlines()
if line is not None:
bounds = innermost_function_bounds(source, int(line), language)
if bounds is None:
raise RecipeError(f"line {line} is not inside a function")
start, end = bounds[0] - 1, bounds[1]
# function_bounds_at anchors on the declarator; the decorators above it are part of the handler
start = _decorator_start(lines, start, _indent(lines[start]))
body = "\n".join(lines[start:end]) + "\n"
else:
body = extract_function(source, function_name, language)
start = _start_line_of(source, body) - 1
end = start + len(body.splitlines())
header = _python_class_header(lines, start) if language == "python" else ()
masked_lines = mask_comments_and_strings(source, language).splitlines()
mention = re.compile(rf"(?<![\w.]){re.escape(function_name)}(?!\w)")
declaration = re.compile(rf"^\s*(?:async\s+)?(?:def|function|sub)\s+{re.escape(function_name)}\b")
covered = set(range(start, end)) | set(header)
registrations = [
lines[index]
for index, masked in enumerate(masked_lines)
if not (start <= index < end) and index < len(lines) and mention.search(masked) and not declaration.match(masked)
if index not in covered and index < len(lines) and mention.search(masked) and not declaration.match(masked)
]
if header:
body = "\n".join(lines[index] for index in header) + "\n" + body
if not registrations:
return body
return body.rstrip("\n") + "\n\n" + "\n".join(line.strip() for line in registrations) + "\n"
def _indent(line: str) -> int:
return len(line) - len(line.lstrip())
def _python_class_header(lines: list[str], start: int) -> tuple[int, ...]:
"""Line indexes of the class declaration enclosing the handler at ``start``, with its decorators.
Flask-RESTX, Django REST framework and Flask's `MethodView` register the *class*: `@ns.route('/profile')` sits
above `class UserProfile(Resource)`, and the method below it is the handler. An excerpt that starts at `def get`
is both unreachable and unparsable (a bare indented block), so the class header travels with it."""
if start >= len(lines) or _indent(lines[start]) == 0:
return ()
indent = _indent(lines[start])
for index in range(start - 1, -1, -1):
match = _CLASS_RE.match(lines[index])
if match is None or len(match.group(1)) >= indent:
continue
return tuple(range(_decorator_start(lines, index, len(match.group(1))), index + 1))
return ()
def innermost_function_bounds(source: str, line: int, language: str = "c") -> tuple[int, int] | None:
"""1-based inclusive bounds of the *innermost* function containing ``line``.
`function_bounds_at` returns the outermost top-level brace group, which is right for C and wrong for the JavaScript
and TypeScript the absence corpus is made of: a handler declared inside a React component or a route module comes
back as the whole 6000-line component, and the excerpt starts mid-parameter-list. Only the handler-context mode
uses this, so hunk and enclosing rows keep the bounds they were harvested with."""
if language == "python":
return function_bounds_at(source, line, language)
masked = mask_comments_and_strings(source, language)
offsets = _line_offsets(source)
if line < 1 or line > len(offsets):
return None
target = offsets[line - 1]
stack: list[int] = []
best: tuple[int, int] | None = None
for pos, ch in enumerate(masked):
if ch == "{":
stack.append(pos)
elif ch == "}" and stack:
open_at = stack.pop()
if pos < target:
continue
start = _declarator_start(source, masked, open_at)
if not (start <= target <= pos):
continue # the declarator line counts: `const f = async () => {` opens its brace after the anchor
head = masked[start:open_at]
if "(" not in head or ")" not in head or _CONTROL_HEAD.match(head.strip()):
continue # an object literal, a bare block, or `if (...) {` — not a callable
if head.count("{") != head.count("}"):
continue # the declarator walk crossed an opening brace: this is a block inside the function, not the function
span = (_line_of(offsets, start), _line_of(offsets, pos))
if best is None or span[0] > best[0]:
best = span
return best
def extract_registration_context(source: str, function_name: str, language: str = "c") -> str:
"""The part of a *separate* document that registers ``function_name``, or "" when it names it nowhere.
Some frameworks keep the route table out of the handler file: connexion registers by `operationId` in an OpenAPI
document, an Express application registers in a router module. That statement carries the guard — the `security`
block, the authorization middleware — so an absence row harvested without it cannot be judged at all. Comment and
string mentions never count. Indented documents return the enclosing block with its ancestor headings; brace
documents return the whole enclosing call, which is where the middleware list lives."""
masked_lines = mask_comments_and_strings(source, language).splitlines()
lines = source.splitlines()
mention = re.compile(rf"(?<![\w.])(?:\w+\.)*{re.escape(function_name)}(?!\w)")
hits = [index for index, masked in enumerate(masked_lines) if mention.search(masked)]
if not hits:
return ""
if language in _INDENTED:
return _indented_block(lines, hits[0])
return _enclosing_call(source, masked_lines, lines, hits[0], function_name)
def _indented_block(lines: list[str], hit: int) -> str:
"""The innermost block containing ``hit``, prefixed by the heading line of every ancestor above it."""
indent = _indent(lines[hit])
ancestors: list[int] = []
current = indent
for index in range(hit - 1, -1, -1):
if not lines[index].strip():
continue
if _indent(lines[index]) < current:
ancestors.append(index)
current = _indent(lines[index])
if current == 0:
break
if not ancestors:
return lines[hit] + "\n"
inner = ancestors[0]
end = inner + 1
while end < len(lines) and (not lines[end].strip() or _indent(lines[end]) > _indent(lines[inner])):
end += 1
body = [*reversed(ancestors[1:]), *range(inner, end)]
return "\n".join(lines[index] for index in body).rstrip("\n") + "\n"
def _enclosing_call(source: str, masked_lines: list[str], lines: list[str], hit: int, function_name: str) -> str:
"""The whole call statement around the mention: `webRouter.post(` … `)` spans four lines and three of them are the guard."""
masked = "\n".join(masked_lines)
offsets = _line_offsets(masked + "\n")
at = masked.find(function_name, offsets[hit])
if at < 0:
return lines[hit] + "\n"
depth = 0
open_at = -1
for pos in range(at - 1, -1, -1):
ch = masked[pos]
if ch == ")":
depth += 1
elif ch == "(":
if depth == 0:
open_at = pos
break
depth -= 1
if open_at < 0:
return lines[hit] + "\n"
depth = 0
close_at = len(masked) - 1
for pos in range(open_at, len(masked)):
if masked[pos] == "(":
depth += 1
elif masked[pos] == ")":
depth -= 1
if depth == 0:
close_at = pos
break
start = _line_of(offsets, open_at) - 1
end = _line_of(offsets, close_at)
return "\n".join(lines[start:end]).rstrip("\n") + "\n"
def extract_line_range(source: str, start: int, end: int) -> str:
"""Inclusive 1-based line range."""
lines = source.splitlines(keepends=True)
@@ -371,7 +572,7 @@ def _python_bounds_at(lines: list[str], line: int) -> tuple[int, int] | None:
# --- recipes and IO ----------------------------------------------------------
def provenance_header(recipe: dict[str, Any], *, side: str, sha: str, upstream_start: int | None = None) -> str:
def provenance_header(recipe: dict[str, Any], *, side: str, sha: str, upstream_start: int | None = None, relpath: str | None = None) -> str:
lines = [
f"Provenance: {recipe.get('repo', '')} {recipe.get('function', '')} ({side}).",
f"repo: {recipe.get('repo', '')}",
@@ -387,7 +588,9 @@ def provenance_header(recipe: dict[str, Any], *, side: str, sha: str, upstream_s
]
if upstream_start is not None:
lines.append(f"upstream_start: {upstream_start}") # 1-based line of the excerpt body in the upstream file
language = language_for(recipe)
if relpath is not None:
lines[8] = f"relpath: {relpath}" # a context document names itself, not the handler file
language = language_for(recipe) if relpath is None else _SUFFIX_LANGUAGE.get(Path(relpath).suffix.lower(), "c")
if language in _HASH_COMMENT:
return "\n".join(f"# {line}" for line in lines) + "\n"
if language in {"cpp", "java", "javascript", "typescript"}:
@@ -414,6 +617,10 @@ def validate_recipe(recipe: dict[str, Any], *, require_license: bool = True) ->
raise RecipeError(f"{name}: enclosing mode needs an integer line")
if require_license and not recipe.get("license"):
raise RecipeError(f"{name}: license required")
for relpath in context_relpaths(recipe):
suffix = Path(relpath).suffix.lower()
if _SUFFIX_LANGUAGE.get(suffix) not in _CONTEXT_LANGUAGES:
raise RecipeError(f"{name}: context document {relpath} has no supported registration reader")
def github_raw_url(repo: str, sha: str, path: str) -> str:
@@ -427,6 +634,22 @@ def excerpt_paths(recipe: dict[str, Any], root: Path) -> tuple[Path, Path]:
return folder / f"vuln{ext}", folder / f"fixed{ext}"
def context_relpaths(recipe: dict[str, Any]) -> tuple[str, ...]:
"""Repo-relative documents that register the handler, fetched at the same two commits as the excerpt."""
value = recipe.get("context") or ()
if isinstance(value, str):
return (value,)
return tuple(str(item) for item in value)
def context_paths(recipe: dict[str, Any], root: Path) -> tuple[tuple[str, Path, Path], ...]:
"""`(relpath, vuln document, fixed document)` per context document. Each side keeps its own copy: the route table
is part of the change often enough (a guard added to the router, not to the handler) that sharing one would hide
the fix."""
folder = root / str(recipe["name"]) / "context"
return tuple((relpath, folder / "vuln" / relpath, folder / "fixed" / relpath) for relpath in context_relpaths(recipe))
_TOKEN_PATTERNS: tuple[tuple[re.Pattern[str], str], ...] = (
(re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----.*?-----END [A-Z ]*PRIVATE KEY-----", re.DOTALL), "***REDACTED***"),
(re.compile(r"(Bearer\s+)[A-Za-z0-9._\-]{12,}"), r"\1***REDACTED***"),
@@ -452,9 +675,18 @@ def redact(body: str) -> str:
return body
def write_excerpt(path: Path, recipe: dict[str, Any], *, side: str, sha: str, body: str, upstream_start: int | None = None) -> None:
def write_excerpt(
path: Path,
recipe: dict[str, Any],
*,
side: str,
sha: str,
body: str,
upstream_start: int | None = None,
relpath: str | None = None,
) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
header = provenance_header(recipe, side=side, sha=sha, upstream_start=upstream_start)
header = provenance_header(recipe, side=side, sha=sha, upstream_start=upstream_start, relpath=relpath)
path.write_text(header + "\n" + redact(body), encoding="utf-8")
@@ -478,12 +710,30 @@ def extract_pair_with_lines(recipe: dict[str, Any], parent_src: str, fix_src: st
raise RecipeError(f"{recipe.get('name')}: line {line}/{fix_line} is not inside a function")
return (extract_line_range(parent_src, *parent_bounds), parent_bounds[0]), (extract_line_range(fix_src, *fix_bounds), fix_bounds[0])
function = str(recipe["function"])
extract = extract_handler_context if mode == "handler_context" else extract_function
parent_text = extract(parent_src, function, language)
fix_text = extract(fix_src, function, language)
if mode == "handler_context":
parent_text, fix_text = _handler_context_pair(recipe, parent_src, fix_src, language)
else:
parent_text, fix_text = extract_function(parent_src, function, language), extract_function(fix_src, function, language)
return (parent_text, _start_line_of(parent_src, parent_text)), (fix_text, _start_line_of(fix_src, fix_text))
def _handler_context_pair(recipe: dict[str, Any], parent_src: str, fix_src: str, language: str) -> tuple[str, str]:
"""Both sides with their registration. ``line``/``fix_line`` anchor when the name repeats or when the fixed twin is
a different, correctly guarded handler from the same snapshot (a Real-Vuln-Benchmark trap): then ``fix_function``
names what the fixed side actually contains."""
line = recipe.get("line")
fix_line = recipe.get("fix_line", line)
return (
extract_handler_context(parent_src, str(recipe["function"]), language, line=int(line) if isinstance(line, int) else None),
extract_handler_context(
fix_src,
str(recipe.get("fix_function") or recipe["function"]),
language,
line=int(fix_line) if isinstance(fix_line, int) else None,
),
)
def _start_line_of(source: str, excerpt: str) -> int:
first = excerpt.splitlines()[0] if excerpt.strip() else ""
for index, line in enumerate(source.splitlines(), start=1):
@@ -534,24 +784,45 @@ def extract_pair(recipe: dict[str, Any], parent_src: str, fix_src: str) -> tuple
raise RecipeError(f"{recipe.get('name')}: line {line}/{fix_line} is not inside a function")
return extract_line_range(parent_src, *parent_bounds), extract_line_range(fix_src, *fix_bounds)
function = str(recipe["function"])
extract = extract_handler_context if mode == "handler_context" else extract_function
return extract(parent_src, function, language), extract(fix_src, function, language)
if mode == "handler_context":
return _handler_context_pair(recipe, parent_src, fix_src, language)
return extract_function(parent_src, function, language), extract_function(fix_src, function, language)
def harvest_recipe(recipe: dict[str, Any], root: Path, *, require_license: bool = True) -> tuple[Path, Path]:
"""Fetch every blob the recipe names, extract, and only then write.
Every fetch happens before the first write on purpose: a re-harvest that half succeeds would leave a reviewed
excerpt truncated and its context document stale, and nothing downstream could tell that apart from a real
upstream change (learning-harness Req 10.2)."""
validate_recipe(recipe, require_license=require_license)
if str(recipe.get("host", "github")) != "github":
raise RecipeError(f"{recipe.get('name')}: live fetch currently implemented for host=github only")
parent_src = fetch_url(github_raw_url(str(recipe["repo"]), str(recipe["parent"]), str(recipe["path"])))
repo, parent, commit = str(recipe["repo"]), str(recipe["parent"]), str(recipe["commit"])
fix_repo = str(recipe.get("fix_repo", repo))
parent_src = fetch_url(github_raw_url(repo, parent, str(recipe["path"])))
# fix_path lets a pair take its fixed side from another file of the same snapshot
# (Real-Vuln-Benchmark false-positive traps are the fixed twins of vulnerable findings).
fix_src = fetch_url(
github_raw_url(str(recipe.get("fix_repo", recipe["repo"])), str(recipe["commit"]), str(recipe.get("fix_path", recipe["path"])))
)
fix_src = fetch_url(github_raw_url(fix_repo, commit, str(recipe.get("fix_path", recipe["path"]))))
function = str(recipe.get("function", ""))
fix_function = str(recipe.get("fix_function") or function)
documents: list[tuple[Path, str, str, str, str]] = []
for relpath, vuln_doc, fixed_doc in context_paths(recipe, root):
language = _SUFFIX_LANGUAGE.get(Path(relpath).suffix.lower(), "c")
for path, side, sha, wanted, source in (
(vuln_doc, "vuln", parent, function, fetch_url(github_raw_url(repo, parent, relpath))),
(fixed_doc, "fixed", commit, fix_function, fetch_url(github_raw_url(fix_repo, commit, relpath))),
):
stanza = extract_registration_context(source, wanted, language)
if not stanza.strip():
raise RecipeError(f"{recipe.get('name')}: context document {relpath} never names {wanted}")
documents.append((path, side, sha, stanza, relpath))
(parent_fn, parent_start), (fix_fn, fix_start) = extract_pair_with_lines(recipe, parent_src, fix_src)
vuln_path, fixed_path = excerpt_paths(recipe, root)
write_excerpt(vuln_path, recipe, side="vuln", sha=str(recipe["parent"]), body=parent_fn, upstream_start=parent_start)
write_excerpt(fixed_path, recipe, side="fixed", sha=str(recipe["commit"]), body=fix_fn, upstream_start=fix_start)
write_excerpt(vuln_path, recipe, side="vuln", sha=parent, body=parent_fn, upstream_start=parent_start)
write_excerpt(fixed_path, recipe, side="fixed", sha=commit, body=fix_fn, upstream_start=fix_start)
for path, side, sha, stanza, relpath in documents:
write_excerpt(path, recipe, side=side, sha=sha, body=stanza, relpath=relpath)
return vuln_path, fixed_path
@@ -599,6 +870,7 @@ def main(argv: list[str] | None = None) -> int:
parser.add_argument("--fetch", action="store_true", help="HTTP-fetch and write excerpts; default is dry-run validate")
parser.add_argument("--force", action="store_true", help="overwrite existing excerpts")
parser.add_argument("--out", type=Path, default=None, help="excerpt root (default: <slice>/)")
parser.add_argument("--report", type=Path, default=None, help="write a JSON row per recipe with its status and reason")
args = parser.parse_args(argv)
slice_root = ROOT / args.slice
recipes_path = args.recipes or slice_root / "recipes.toml"
@@ -619,28 +891,38 @@ def main(argv: list[str] | None = None) -> int:
print("pass --name RECIPE, --prefix PREFIX, or --all", file=sys.stderr)
return 2
failures = 0
report: list[dict[str, str]] = []
for recipe in selected:
name = str(recipe["name"])
try:
validate_recipe(recipe)
except RecipeError as exc:
print(str(exc), file=sys.stderr)
report.append({"name": name, "status": "failed", "reason": str(exc)})
failures += 1
continue
if not args.fetch:
print(f"ok {name} parent={recipe['parent']} commit={recipe['commit']} mode={recipe.get('mode', 'name')}")
report.append({"name": name, "status": "validated", "reason": ""})
continue
vuln_path, fixed_path = excerpt_paths(recipe, out)
if vuln_path.is_file() and fixed_path.is_file() and not args.force:
print(f"skip {name} (exists)")
report.append({"name": name, "status": "skipped", "reason": "excerpt already present"})
continue
try:
harvest_recipe(recipe, out)
except (RecipeError, OSError) as exc:
# The excerpt on disk is the reviewed one; a fetch that failed leaves it exactly as it was and says why.
print(f"{name}: {exc}", file=sys.stderr)
report.append({"name": name, "status": "failed", "reason": str(exc)})
failures += 1
continue
print(f"wrote {name} -> {vuln_path.parent}")
report.append({"name": name, "status": "written", "reason": ""})
if args.report is not None:
args.report.parent.mkdir(parents=True, exist_ok=True)
args.report.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
return 1 if failures else 0
benchmarks/pairs/vibe-py/catalog.toml
+14
View File
@@ -3272,6 +3272,7 @@ class = "broken_access_control"
path = "server/api/v1/api_v1.py"
function = "delete"
mechanism = "missing_auth_guard"
family = "access_control"
evidence = "BFLA (Broken Function Level Authorization) on the delete-user API endpoint. The @token_required decorator is commented out (lines 221-222), so DELETE /api/v1/delete-user/<user_id> can be called by anyone without authentication to delete any"
[[pair]]
@@ -3301,6 +3302,7 @@ class = "broken_access_control"
path = "server/api/v1/api_v1.py"
function = "get"
mechanism = "missing_auth_guard"
family = "access_control"
evidence = "BOLA (Broken Object Level Authorization) on the API profile endpoint. The @token_required decorator is commented out (line 152-153), so /api/v1/profile?user_id=X returns any user's profile (username, email, country, role, permissions, team)"
[[pair]]
@@ -3353,12 +3355,18 @@ provenance = "human"
split = "holdout"
review_tier = "seeded"
[[pair.context]]
relpath = "openapi_specs/openapi3.yml"
vuln = "vampi-books-get-by-title/context/vuln/openapi_specs/openapi3.yml"
fixed = "vampi-books-get-by-title/context/fixed/openapi_specs/openapi3.yml"
[[pair.expected]]
cwe = "CWE-639"
class = "broken_access_control"
path = "api_views/books.py"
function = "get_by_title"
mechanism = "missing_auth_guard"
family = "access_control"
evidence = "Broken Object Level Authorization (BOLA). When vuln=1, get_by_title queries the book by title only (Book.query.filter_by(book_title=str(book_title)).first()) without verifying the authenticated user is the owner. Any authenticated user can "
[[pair]]
@@ -3411,12 +3419,18 @@ provenance = "human"
split = "holdout"
review_tier = "seeded"
[[pair.context]]
relpath = "openapi_specs/openapi3.yml"
vuln = "vampi-users-update-password/context/vuln/openapi_specs/openapi3.yml"
fixed = "vampi-users-update-password/context/fixed/openapi_specs/openapi3.yml"
[[pair.expected]]
cwe = "CWE-639"
class = "broken_access_control"
path = "api_views/users.py"
function = "update_password"
mechanism = "missing_auth_guard"
family = "access_control"
evidence = "Unauthorized password change (IDOR). When vuln=1, the update_password function uses the username from the URL path parameter to look up the user whose password to change, instead of using the authenticated user from the JWT token (resp['sub"
[[pair]]
benchmarks/pairs/vibe-py/recipes.toml
+13 -4
View File
@@ -3208,7 +3208,7 @@ parent = "c4db46f72586c4ecb65859051ea037018dbf0e5e"
commit = "c4db46f72586c4ecb65859051ea037018dbf0e5e"
path = "server/api/v1/api_v1.py"
fix_path = "server/api/v1/api_v1.py"
mode = "enclosing"
mode = "handler_context"
line = 223
fix_line = 123
function = "delete"
@@ -3229,6 +3229,8 @@ trap_id = "threatbyte-fp-004"
acceptable_cwes = ["CWE-862", "CWE-306", "CWE-284", "CWE-287"]
evidence = "BFLA (Broken Function Level Authorization) on the delete-user API endpoint. The @token_required decorator is commented out (lines 221-222), so DELETE /api/v1/delete-user/<user_id> can be called by anyone without authentication to delete any"
commit_url = "https://github.com/anotherik/ThreatByte/blob/c4db46f72586c4ecb65859051ea037018dbf0e5e/server/api/v1/api_v1.py#L223"
family = "access_control"
fix_function = "post"
split = "train"
[[recipe]]
@@ -3239,7 +3241,7 @@ parent = "c4db46f72586c4ecb65859051ea037018dbf0e5e"
commit = "c4db46f72586c4ecb65859051ea037018dbf0e5e"
path = "server/api/v1/api_v1.py"
fix_path = "server/api/v1/api_v1.py"
mode = "enclosing"
mode = "handler_context"
line = 154
fix_line = 123
function = "get"
@@ -3260,6 +3262,8 @@ trap_id = "threatbyte-fp-004"
acceptable_cwes = ["CWE-639", "CWE-284", "CWE-862", "CWE-863"]
evidence = "BOLA (Broken Object Level Authorization) on the API profile endpoint. The @token_required decorator is commented out (line 152-153), so /api/v1/profile?user_id=X returns any user's profile (username, email, country, role, permissions, team)"
commit_url = "https://github.com/anotherik/ThreatByte/blob/c4db46f72586c4ecb65859051ea037018dbf0e5e/server/api/v1/api_v1.py#L154"
family = "access_control"
fix_function = "post"
split = "holdout"
[[recipe]]
@@ -3301,7 +3305,7 @@ parent = "1713b54b601ad29582581eeda4b31fceb1319874"
commit = "1713b54b601ad29582581eeda4b31fceb1319874"
path = "api_views/books.py"
fix_path = "api_views/books.py"
mode = "enclosing"
mode = "handler_context"
line = 51
fix_line = 62
function = "get_by_title"
@@ -3322,6 +3326,8 @@ trap_id = "vampi-fp-003"
acceptable_cwes = ["CWE-639", "CWE-284", "CWE-285", "CWE-862"]
evidence = "Broken Object Level Authorization (BOLA). When vuln=1, get_by_title queries the book by title only (Book.query.filter_by(book_title=str(book_title)).first()) without verifying the authenticated user is the owner. Any authenticated user can "
commit_url = "https://github.com/erev0s/VAmPI/blob/1713b54b601ad29582581eeda4b31fceb1319874/api_views/books.py#L51"
family = "access_control"
context = ["openapi_specs/openapi3.yml"]
split = "holdout"
[[recipe]]
@@ -3363,7 +3369,7 @@ parent = "1713b54b601ad29582581eeda4b31fceb1319874"
commit = "1713b54b601ad29582581eeda4b31fceb1319874"
path = "api_views/users.py"
fix_path = "api_views/users.py"
mode = "enclosing"
mode = "handler_context"
line = 187
fix_line = 55
function = "update_password"
@@ -3384,6 +3390,9 @@ trap_id = "vampi-fp-001"
acceptable_cwes = ["CWE-639", "CWE-284", "CWE-285", "CWE-862"]
evidence = "Unauthorized password change (IDOR). When vuln=1, the update_password function uses the username from the URL path parameter to look up the user whose password to change, instead of using the authenticated user from the JWT token (resp['sub"
commit_url = "https://github.com/erev0s/VAmPI/blob/1713b54b601ad29582581eeda4b31fceb1319874/api_views/users.py#L187"
family = "access_control"
fix_function = "register_user"
context = ["openapi_specs/openapi3.yml"]
split = "holdout"
[[recipe]]
benchmarks/pairs/vibe-py/threatbyte-api-v1-delete/fixed.py
+8
View File
@@ -9,7 +9,15 @@
# relpath: server/api/v1/api_v1.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 103
@ns.route('/login')
class UserLogin(Resource):
@ns.expect(login_model)
@ns.response(200, 'Login successful', login_response_model)
@ns.response(400, 'Please provide username and password')
@ns.response(401, 'Invalid username or password')
@ns.response(405, 'Method not allowed')
def post(self):
"""
Handles user login.
benchmarks/pairs/vibe-py/threatbyte-api-v1-delete/vuln.py
+9
View File
@@ -9,7 +9,16 @@
# relpath: server/api/v1/api_v1.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 215
@ns.route('/delete-user/<int:user_id>')
@ns.doc(description='Delete a user from the database without proper authorization checks. '
'This endpoint represents a Broken Function Level Authorization vulnerability.',
responses={200: ('User successfully deleted', delete_model),
404: ('User not found', error_model)})
class UserDelete(Resource):
#@token_required
#def delete(self, current_user, user_id):
def delete(self, user_id):
"""
Delete a user from the application.
benchmarks/pairs/vibe-py/threatbyte-api-v1-get/fixed.py
+8
View File
@@ -9,7 +9,15 @@
# relpath: server/api/v1/api_v1.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 103
@ns.route('/login')
class UserLogin(Resource):
@ns.expect(login_model)
@ns.response(200, 'Login successful', login_response_model)
@ns.response(400, 'Please provide username and password')
@ns.response(401, 'Invalid username or password')
@ns.response(405, 'Method not allowed')
def post(self):
"""
Handles user login.
benchmarks/pairs/vibe-py/threatbyte-api-v1-get/vuln.py
+9
View File
@@ -9,7 +9,16 @@
# relpath: server/api/v1/api_v1.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 146
@ns.route('/profile')
class UserProfile(Resource):
@ns.expect(profile_query)
@ns.response(200, 'Success', user_profile_model)
@ns.response(400, 'User ID is required')
@ns.response(404, 'User profile not found')
#@token_required
#def get(self, current_user):
def get(self):
"""
Retrieves the profile information of a user based on the provided user ID.
benchmarks/pairs/vibe-py/vampi-books-get-by-title/context/fixed/openapi_specs/openapi3.yml
+79
View File
@@ -0,0 +1,79 @@
# Provenance: erev0s/VAmPI get_by_title (fixed).
# repo: erev0s/VAmPI
# commit: 1713b54b601ad29582581eeda4b31fceb1319874
# parent: 1713b54b601ad29582581eeda4b31fceb1319874
# commit_url: https://github.com/erev0s/VAmPI/blob/1713b54b601ad29582581eeda4b31fceb1319874/api_views/books.py#L51
# cve:
# license: MIT
# function: get_by_title
# relpath: openapi_specs/openapi3.yml
# provenance: human
# mechanism: missing_auth_guard
paths:
/books/v1/{book_title}:
get:
security:
- bearerAuth: [ ]
tags:
- books
summary: Retrieves book by title along with secret
description: >-
Retrieves book by title along with secret. Only the owner may retrieve
it
operationId: api_views.books.get_by_title
parameters:
- name: book_title
in: path
description: retrieve book data
required: true
schema:
type: string
example: 'bookTitle77'
responses:
'200':
description: Successfully retrieve book info
content:
application/json:
schema:
type: array
items:
type: object
properties:
book_title:
type: string
example: 'bookTitle77'
owner:
type: string
example: 'name1'
secret:
type: string
example: 'secret for bookTitle77'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid token. Please log in again.'
'404':
description: Book not found
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Book not found!'
benchmarks/pairs/vibe-py/vampi-books-get-by-title/context/vuln/openapi_specs/openapi3.yml
+79
View File
@@ -0,0 +1,79 @@
# Provenance: erev0s/VAmPI get_by_title (vuln).
# repo: erev0s/VAmPI
# commit: 1713b54b601ad29582581eeda4b31fceb1319874
# parent: 1713b54b601ad29582581eeda4b31fceb1319874
# commit_url: https://github.com/erev0s/VAmPI/blob/1713b54b601ad29582581eeda4b31fceb1319874/api_views/books.py#L51
# cve:
# license: MIT
# function: get_by_title
# relpath: openapi_specs/openapi3.yml
# provenance: human
# mechanism: missing_auth_guard
paths:
/books/v1/{book_title}:
get:
security:
- bearerAuth: [ ]
tags:
- books
summary: Retrieves book by title along with secret
description: >-
Retrieves book by title along with secret. Only the owner may retrieve
it
operationId: api_views.books.get_by_title
parameters:
- name: book_title
in: path
description: retrieve book data
required: true
schema:
type: string
example: 'bookTitle77'
responses:
'200':
description: Successfully retrieve book info
content:
application/json:
schema:
type: array
items:
type: object
properties:
book_title:
type: string
example: 'bookTitle77'
owner:
type: string
example: 'name1'
secret:
type: string
example: 'secret for bookTitle77'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid token. Please log in again.'
'404':
description: Book not found
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Book not found!'
benchmarks/pairs/vibe-py/vampi-books-get-by-title/fixed.py
+1
View File
@@ -9,6 +9,7 @@
# relpath: api_views/books.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 45
def get_by_title(book_title):
resp = token_validator(request.headers.get('Authorization'))
benchmarks/pairs/vibe-py/vampi-books-get-by-title/vuln.py
+1
View File
@@ -9,6 +9,7 @@
# relpath: api_views/books.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 45
def get_by_title(book_title):
resp = token_validator(request.headers.get('Authorization'))
benchmarks/pairs/vibe-py/vampi-users-update-password/context/fixed/openapi_specs/openapi3.yml
+55
View File
@@ -0,0 +1,55 @@
# Provenance: erev0s/VAmPI update_password (fixed).
# repo: erev0s/VAmPI
# commit: 1713b54b601ad29582581eeda4b31fceb1319874
# parent: 1713b54b601ad29582581eeda4b31fceb1319874
# commit_url: https://github.com/erev0s/VAmPI/blob/1713b54b601ad29582581eeda4b31fceb1319874/api_views/users.py#L187
# cve:
# license: MIT
# function: update_password
# relpath: openapi_specs/openapi3.yml
# provenance: human
# mechanism: missing_auth_guard
paths:
/users/v1/register:
post:
tags:
- users
summary: Register new user
description: Register new user
operationId: api_views.users.register_user
requestBody:
description: Username of the user
content:
application/json:
schema:
type: object
properties:
username:
type: string
example: 'name1'
password:
type: string
example: 'pass1'
email:
type: string
example: '[email protected]'
required: true
responses:
'200':
description: Successfully created user
content:
application/json:
schema:
type: object
properties:
message:
type: string
example: 'Successfully registered. Login to receive an auth token.'
status:
type: string
enum: ['success', 'fail']
example: 'success'
'400':
description: Invalid request
content: {}
benchmarks/pairs/vibe-py/vampi-users-update-password/context/vuln/openapi_specs/openapi3.yml
+73
View File
@@ -0,0 +1,73 @@
# Provenance: erev0s/VAmPI update_password (vuln).
# repo: erev0s/VAmPI
# commit: 1713b54b601ad29582581eeda4b31fceb1319874
# parent: 1713b54b601ad29582581eeda4b31fceb1319874
# commit_url: https://github.com/erev0s/VAmPI/blob/1713b54b601ad29582581eeda4b31fceb1319874/api_views/users.py#L187
# cve:
# license: MIT
# function: update_password
# relpath: openapi_specs/openapi3.yml
# provenance: human
# mechanism: missing_auth_guard
paths:
/users/v1/{username}/password:
put:
security:
- bearerAuth: [ ]
tags:
- users
summary: Update users password
description: Update users password
operationId: api_views.users.update_password
parameters:
- name: username
in: path
description: username to update password
required: true
schema:
type: string
example: 'name1'
requestBody:
description: field to update
content:
application/json:
schema:
type: object
properties:
password:
type: string
example: 'pass4'
required: true
responses:
'204':
description: Successfully updated users password
content: {}
'400':
description: Invalid request
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Malformed Data'
'401':
description: User not authorized
content:
application/json:
schema:
type: object
properties:
status:
type: string
enum: ['fail']
example: 'fail'
message:
type: string
example: 'Invalid token. Please log in again.'
benchmarks/pairs/vibe-py/vampi-users-update-password/fixed.py
+1
View File
@@ -9,6 +9,7 @@
# relpath: api_views/users.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 52
def register_user():
request_data = request.get_json()
benchmarks/pairs/vibe-py/vampi-users-update-password/vuln.py
+1
View File
@@ -9,6 +9,7 @@
# relpath: api_views/users.py
# provenance: human
# mechanism: missing_auth_guard
# upstream_start: 179
def update_password(username):
request_data = request.get_json()
src/openultrasast/learning/rounds.py
+3 -3
View File
@@ -125,7 +125,7 @@ def score_case(case: PairCase, scan: Scan, *, taxonomy: FamilyTaxonomy, runs: in
A pair the corpus already marked unscorable is never run: spending model calls on a row that cannot
be scored buys nothing and would put its noise into a floor it does not belong in.
"""
from ..pairs import _materialize, _overlay_scan
from ..pairs import _materialize_side, _overlay_scan
name = family or _family_of(case, taxonomy)
if case.unscorable:
@@ -138,8 +138,8 @@ def score_case(case: PairCase, scan: Scan, *, taxonomy: FamilyTaxonomy, runs: in
unscorable_reason=case.unscorable,
)
with tempfile.TemporaryDirectory(prefix="ousast-round-") as scratch:
vuln_root = _materialize(Path(scratch) / "vuln", case.vuln_file, case.relpath)
fix_root = _materialize(Path(scratch) / "fixed", case.fixed_file, case.relpath)
vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
ranges = _overlay_scan(vuln_root).ranges
vuln_runs = [scan(vuln_root) for _ in range(max(runs, 1))]
fix_runs = [scan(fix_root) for _ in range(max(runs, 1))]
src/openultrasast/mapping.py
+126 -8
View File
@@ -105,6 +105,7 @@ def analyze_entry_points(root: Path, targets: list[FileTarget]) -> list[EntryPoi
text = path.read_text(errors="ignore")
records.extend(_language_entry_points(target, text))
records.extend(_tag_entry_points(target))
records.extend(_route_manifest_entry_points(root, targets))
return sorted(records, key=lambda item: (item.path, item.line is None, item.line or 0, item.name))
@@ -249,9 +250,8 @@ def _js_entry_points(target: FileTarget, text: str) -> list[EntryPointRecord]:
route_tokens = (".get(", ".post(", ".put(", ".delete(", ".patch(", ".use(")
lines = text.splitlines()
bounds = _js_function_bounds(lines)
for number, line in enumerate(lines, start=1):
stripped = line.strip()
if any(token in stripped for token in route_tokens) and any(prefix in stripped for prefix in ("app", "router", "server")):
for number, stripped in _js_statements(lines):
if any(token in stripped for token in route_tokens) and any(prefix in stripped.lower() for prefix in ("app", "router", "server")):
handler, middleware = _js_registration(stripped, bounds)
access, evidence = _js_route_access(middleware, stripped)
start, end = bounds.get(handler or "", (number, number))
@@ -288,6 +288,89 @@ def _js_entry_points(target: FileTarget, text: str) -> list[EntryPointRecord]:
return records
_OPERATION_ID = re.compile(r"""^\s*(?:-\s*)?["']?operationId["']?\s*:\s*["']?([\w.]+)["']?,?\s*$""")
_MANIFEST_SUFFIXES = (".yml", ".yaml", ".json")
_MANIFEST_MAX_BYTES = 2_000_000
def _route_manifest_entry_points(root: Path, targets: Sequence[FileTarget]) -> list[EntryPointRecord]:
"""Handlers a *document* registers: an OpenAPI `operationId` names the function connexion, FastAPI-from-spec or an
API gateway will call, and the `security` block beside it is the guard.
Frameworks that keep the route table out of the handler file leave every handler looking like a plain function, so
an absence bug on one is unreachable and unjudgeable. The document itself is not a source file — it never becomes a
file target — so the record is attributed to the module the operation id points at when that module is present
(learning-harness Req 10.2)."""
modules = {target.path for target in targets}
records: list[EntryPointRecord] = []
for path in sorted(root.rglob("*")):
if not path.is_file() or path.suffix.lower() not in _MANIFEST_SUFFIXES:
continue
try:
if path.stat().st_size > _MANIFEST_MAX_BYTES:
continue
lines = path.read_text(errors="ignore").splitlines()
except OSError:
continue
relative = path.relative_to(root).as_posix()
for index, line in enumerate(lines):
match = _OPERATION_ID.match(line)
if match is None:
continue
operation = match.group(1)
handler = operation.rsplit(".", 1)[-1]
owner = _operation_module(operation, modules)
access, evidence = _manifest_access(lines, index)
records.append(
EntryPointRecord(
path=owner or relative,
line=None if owner else index + 1,
end_line=None if owner else index + 1,
function_name=handler,
name=operation,
kind="route",
access_level=access,
trust_boundary="http_request",
access_evidence=evidence,
conditions=[],
provenance=f"entrypoint:route:{access}",
rationale=f"registered by operationId {operation} in {relative}",
)
)
return records
def _operation_module(operation: str, modules: set[str]) -> str | None:
"""`api_views.books.get_by_title` -> `api_views/books.py` when that file is in the snapshot."""
parts = operation.split(".")
if len(parts) < 2:
return None
for depth in range(len(parts) - 1, 0, -1):
candidate = "/".join(parts[:depth]) + ".py"
if candidate in modules:
return candidate
return None
def _manifest_access(lines: Sequence[str], hit: int) -> tuple[AccessLevel, list[str]]:
"""The operation's own `security` block decides. `security: []` is an explicit opt-out and stays public."""
indent = len(lines[hit]) - len(lines[hit].lstrip())
start = hit
while start > 0 and (not lines[start - 1].strip() or len(lines[start - 1]) - len(lines[start - 1].lstrip()) >= indent):
start -= 1
end = hit + 1
while end < len(lines) and (not lines[end].strip() or len(lines[end]) - len(lines[end].lstrip()) >= indent):
end += 1
for index in range(start, end):
stripped = lines[index].strip()
if not stripped.startswith("security"):
continue
if stripped.replace(" ", "") in {"security:[]", '"security":[]', "security:[],", '"security":[],'}:
return "public", [f"{stripped} declares the operation public"]
return "authenticated", [f"security block at {lines[index].strip()}"]
return "public", ["operation declares no security requirement"]
_RESOURCE_BASES = ("resource", "methodview", "apiview", "viewset", "modelviewset", "httpendpoint")
_HTTP_VERBS = ("get", "post", "put", "patch", "delete", "head", "options")
@@ -534,12 +617,21 @@ def _safe_unparse(node: ast.AST) -> str:
return ""
_STRING_LITERAL = re.compile(r"""(['"]).*?\1""", re.DOTALL)
def _python_route_access(decorators: list[str]) -> tuple[AccessLevel, list[str]]:
"""The guard is the decorator's *name*, never its prose.
`@ns.doc(description='... without proper authorization checks')` documents the bug; matching "authorization"
inside it reads an unguarded endpoint as guarded and silences every obligation on it, which is exactly the
absence the corpus exists to catch (learning-harness Req 10.2)."""
role_tokens = ("role", "permission", "admin", "owner")
auth_tokens = ("login_required", "authenticated", "auth", "jwt_required")
auth_tokens = ("login_required", "authenticated", "auth", "jwt_required", "token_required", "require_token")
access_decorators = [decorator for decorator in decorators if not _looks_like_route_decorator(decorator)]
evidence = [decorator for decorator in access_decorators if any(token in decorator.lower() for token in role_tokens + auth_tokens)]
lowered = "\n".join(evidence).lower()
named = {decorator: _STRING_LITERAL.sub("", decorator).lower() for decorator in access_decorators}
evidence = [decorator for decorator in access_decorators if any(token in named[decorator] for token in role_tokens + auth_tokens)]
lowered = "\n".join(named[decorator] for decorator in evidence)
if any(token in lowered for token in role_tokens):
return "role-restricted", evidence
if any(token in lowered for token in auth_tokens):
@@ -593,6 +685,32 @@ def _js_function_bounds(lines: Sequence[str]) -> dict[str, tuple[int, int]]:
return bounds
def _js_statements(lines: Sequence[str]) -> list[tuple[int, str]]:
"""One entry per statement, with the line its first token sits on.
An express route is often written over several lines, one middleware per line, and the guard is exactly what
those middle lines say. Read line by line, `webRouter.post(` carries no handler and no middleware at all, so the
route reads as public and the handler is never named (learning-harness Req 10.2)."""
statements: list[tuple[int, str]] = []
index = 0
while index < len(lines):
stripped = lines[index].strip()
joined, span = stripped, 1
while joined.count("(") > joined.count(")") and span < _MAX_STATEMENT_LINES and index + span < len(lines):
joined = f"{joined} {lines[index + span].strip()}"
span += 1
if span > 1 and joined.count("(") <= joined.count(")"):
statements.append((index + 1, joined))
index += span
continue
statements.append((index + 1, stripped))
index += 1
return statements
_MAX_STATEMENT_LINES = 12
def _js_registration(line: str, bounds: Mapping[str, tuple[int, int]]) -> tuple[str | None, list[str]]:
"""`router.get('/books', requireAuth, listBooks)` -> the handler identifier and the middleware identifiers before it."""
body = line[line.find("(") + 1 : line.rfind(")")] if "(" in line and line.rfind(")") > line.find("(") else ""
@@ -600,9 +718,9 @@ def _js_registration(line: str, bounds: Mapping[str, tuple[int, int]]) -> tuple[
identifiers = [part for part in parts[1:] if _JS_IDENTIFIER.match(part)]
if not identifiers:
return None, [line]
handler = next((name for name in reversed(identifiers) if name in bounds), identifiers[-1])
handler = next((name for name in reversed(identifiers) if name in bounds or name.rsplit(".", 1)[-1] in bounds), identifiers[-1])
middleware = [name for name in identifiers if name != handler]
return handler, middleware
return handler.rsplit(".", 1)[-1], middleware # `ExportsController.exportProject` is registered; `exportProject` is defined
def _js_route_access(middleware: Sequence[str], line: str) -> tuple[AccessLevel, list[str]]:
src/openultrasast/pairs.py
+39 -5
View File
@@ -132,6 +132,10 @@ class PairCase:
split_declared: bool = True # False when the catalog row named no split, so the split helper may assign one
fix_date: str = "" # ISO date of the fixing commit; the catalog generator fills it, the split helper orders by it
recipe: tuple[tuple[str, object], ...] = () # pointer pairs carry their harvest recipe instead of excerpts
# (relpath, vuln document, fixed document): files that register the handler from outside its own module — an
# OpenAPI operation, a router table. Materialized beside the excerpt so the mapper can name the handler at all
# (learning-harness Req 10.2).
context_files: tuple[tuple[str, Path, Path], ...] = ()
@dataclass(frozen=True)
@@ -264,6 +268,7 @@ def _load_catalog_file(path: Path) -> tuple[PairCase, ...]:
vuln_file, fixed_file = _resolve(root, str(item["vuln"])), _resolve(root, str(item["fixed"]))
else:
vuln_file, fixed_file, recipe = _pointer_files(item, slice_name, name)
context_files = _context_files(item, root, name)
cases.append(
PairCase(
name=name,
@@ -291,11 +296,30 @@ def _load_catalog_file(path: Path) -> tuple[PairCase, ...]:
reviewer=reviewer,
vendored=vendored,
recipe=recipe,
context_files=context_files,
)
)
return tuple(cases)
def _context_files(item: dict[str, object], root: Path, name: str) -> tuple[tuple[str, Path, Path], ...]:
rows = item.get("context")
if not rows:
return ()
if not isinstance(rows, list):
raise CatalogError(f"pair {name}: context must be a list of tables")
out: list[tuple[str, Path, Path]] = []
for row in rows:
if not isinstance(row, dict) or not {"relpath", "vuln", "fixed"} <= set(row):
raise CatalogError(f"pair {name}: each context row needs relpath, vuln and fixed")
vuln, fixed = _resolve(root, str(row["vuln"])), _resolve(root, str(row["fixed"]))
for path in (vuln, fixed):
if not path.is_file():
raise CatalogError(f"pair {name}: context document {path} is missing; re-harvest the row")
out.append((str(row["relpath"]), vuln, fixed))
return tuple(out)
def pair_cache_dir() -> Path:
"""Cache root for pointer pairs (Req 10.2); outside the repository unless the operator points it elsewhere."""
return Path(os.environ.get(PAIR_CACHE_ENV) or DEFAULT_PAIR_CACHE).expanduser()
@@ -395,8 +419,8 @@ def _review_tier(item: dict[str, object], slice_name: str, name: str) -> tuple[s
def evaluate_pair(case: PairCase, *, hunter: HunterScan | None = None, ruleset: tuple[PatternRule, ...] | None = None) -> PairOutcome:
with tempfile.TemporaryDirectory(prefix="ousast-pair-") as scratch:
vuln_root = _materialize(Path(scratch) / "vuln", case.vuln_file, case.relpath)
fix_root = _materialize(Path(scratch) / "fixed", case.fixed_file, case.relpath)
vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
if hunter is not None:
return _evaluate_hunter_pair(case, vuln_root, fix_root, hunter)
if case.slice in OVERLAY_SLICES:
@@ -609,7 +633,9 @@ def duplicate_groups(cases: Sequence[PairCase]) -> tuple[tuple[str, ...], ...]:
digest = _body_digest(case.vuln_file) if case.vendored else None
if digest is not None:
by_digest.setdefault(digest, []).append(case.name)
return tuple(tuple(sorted(names)) for _, names in sorted(by_digest.items()) if len(names) > 1)
# Ordered by name, not by digest: the report is a committed artifact and a re-harvest that changes one excerpt
# must not reshuffle rows it did not touch.
return tuple(sorted(tuple(sorted(names)) for names in by_digest.values() if len(names) > 1))
def split_by_repository(cases: Sequence[PairCase], *, holdout_fraction: float = 0.5) -> tuple[tuple[PairCase, ...], SplitReport]:
@@ -783,6 +809,14 @@ def _materialize(dest: Path, source: Path, relpath: str) -> Path:
return dest
def _materialize_side(dest: Path, case: PairCase, *, side: str) -> Path:
"""The excerpt plus the documents that register it, at their own repository-relative paths."""
root = _materialize(dest, case.vuln_file if side == "vuln" else case.fixed_file, case.relpath)
for relpath, vuln_doc, fixed_doc in case.context_files:
_materialize(root, vuln_doc if side == "vuln" else fixed_doc, relpath)
return root
# --- scans -----------------------------------------------------------------
@@ -864,8 +898,8 @@ def _obligation_scan(root: Path, targets: Sequence[FileTarget]) -> list[StaticFi
def _inventory_only(case: PairCase, ruleset: tuple[PatternRule, ...] | None = None) -> PairOutcome:
with tempfile.TemporaryDirectory(prefix="ousast-pair-inv-") as scratch:
vuln_root = _materialize(Path(scratch) / "vuln", case.vuln_file, case.relpath)
fix_root = _materialize(Path(scratch) / "fixed", case.fixed_file, case.relpath)
vuln_root = _materialize_side(Path(scratch) / "vuln", case, side="vuln")
fix_root = _materialize_side(Path(scratch) / "fixed", case, side="fixed")
return _evaluate_inventory_pair(case, vuln_root, fix_root, ruleset=ruleset)
src/openultrasast/preprocess.py
+2
View File
@@ -18,6 +18,8 @@ LANGUAGE_BY_EXTENSION = {
".py": "python",
".js": "javascript",
".jsx": "javascript",
".mjs": "javascript", # ES modules and CommonJS modules are JavaScript; without them the file has no target at all
".cjs": "javascript",
".ts": "typescript",
".tsx": "typescript",
".go": "go",
tests/test_learning_classifier_report.py
+5 -1
View File
@@ -44,7 +44,11 @@ def test_the_report_covers_the_whole_corpus_and_names_the_unknowns(tmp_path: Pat
assert report.unknown == 2 and report.per_tier["static"] == len(cases) - 2 # tier one places all but two rows
payload = report.to_dict()
json.dumps(payload)
assert payload["labeled"] == 0 # no maintainer family labels in the corpus yet, and the report says so
# Task 4.1 declared `family` on the ten absence rows; only the four at a reference tier (`seeded`) count as labels
# a maintainer stands behind, and the six `title`-tier agent rows do not. Agreement over four rows of one family is
# not yet a measurement of the classifier, so the number is recorded, not celebrated.
assert payload["labeled"] == 4 and payload["agreement"] == 1.0
assert set(report.confusion) == {"access_control"}
def test_agreement_uses_the_first_answer_and_the_oracle_uses_any_of_them(tmp_path: Path) -> None:
tests/test_pair_hygiene.py
+17
View File
@@ -75,6 +75,23 @@ def test_duplicate_bodies_across_pairs_are_reported_before_any_split() -> None:
payload = json.loads(report.read_text())
assert payload["duplicate_groups"] == [list(names) for names in groups]
assert set(payload["identical_twins"]) == IDENTICAL_TWINS
assert groups == tuple(sorted(groups)), "the committed report must not reshuffle when one excerpt is re-harvested"
def test_duplicate_groups_that_cross_the_split_are_named_as_the_leak_they_are() -> None:
"""Two byte-identical excerpts on opposite sides of the split teach the holdout row exactly.
`split_by_repository` cannot see this: the duplicates are different repositories (a fork of a project, or one CVE
harvested under two names), so the repository-level split reports no straddle and the leak stays invisible."""
from openultrasast.pairs import duplicate_groups
cases = select_vendored(load_pair_catalog(DEFAULT_CATALOG))
splits = {case.name: case.split for case in cases}
crossing = [list(names) for names in duplicate_groups(cases) if len({splits[name] for name in names}) > 1]
payload = json.loads(Path("benchmarks/measurements/2026-09-06-corpus-hygiene.json").read_text())
assert payload["cross_split_duplicates"] == crossing
assert len(crossing) == 9, "nine of the ten duplicate groups cross the split; the number is the finding"
assert ["openniw-jobs-0fc947", "openniw-jobs-b74de3"] in crossing # two forks of one project, train and holdout
def test_split_by_repository_keeps_declared_splits_and_reports_straddles() -> None:
tests/test_pair_registration_context.py
+427
View File
@@ -0,0 +1,427 @@
"""Absence rows travel with the statements that register them (learning-harness 4.1, Req 10.2).
An access-control bug is the absence of a guard, so the excerpt has to carry the code that says the
handler is reachable at all: its decorators, the class its framework registers, and — when the
registration lives in another document — that document. Without it the entry-point mapper names no
handler, every obligation on the row is unreachable, and the pair scores as ``missing_context``
forever rather than as a detector miss.
"""
from __future__ import annotations
import runpy
import shutil
import tempfile
import tomllib
from pathlib import Path
import pytest
from openultrasast.learning.families import load_families
from openultrasast.mapping import analyze_entry_points
from openultrasast.pairs import DEFAULT_CATALOG, load_pair_catalog, select_vendored
from openultrasast.preprocess import preprocess_repository
LIB = Path("benchmarks/pairs/harvest.py")
FAMILIES = Path("src/openultrasast/ruleset/families.toml")
def _lib() -> dict[str, object]:
return runpy.run_path(str(LIB), run_name="pair_harvest_lib")
# --- extraction --------------------------------------------------------------
RESTX_SOURCE = """from flask_restx import Resource
@ns.route('/profile')
class UserProfile(Resource):
@ns.expect(profile_query)
#@token_required
def get(self):
return dict(User.by_id(request.args['user_id'])), 200
@ns.route('/settings')
class Settings(Resource):
@token_required
def get(self, current_user):
return dict(current_user.settings), 200
"""
def test_handler_context_carries_the_class_its_framework_registers() -> None:
"""A flask-restx handler is a method: the route decorator sits on the class, not the function."""
lib = _lib()
excerpt = lib["extract_handler_context"](RESTX_SOURCE, "get", "python", line=7)
assert "@ns.route('/profile')" in excerpt
assert "class UserProfile(Resource):" in excerpt
assert "def get(self):" in excerpt
assert "class Settings" not in excerpt and "current_user.settings" not in excerpt
compile(excerpt, "<excerpt>", "exec") # a class fragment must still be importable Python
def test_handler_context_anchors_on_the_line_when_the_method_name_repeats() -> None:
"""Two `def get` in one file: the name alone cannot say which handler the row labels."""
lib = _lib()
second = lib["extract_handler_context"](RESTX_SOURCE, "get", "python", line=13)
assert "class Settings(Resource):" in second and "@token_required" in second
assert "UserProfile" not in second
def test_registration_context_returns_the_openapi_operation_stanza() -> None:
"""connexion registers by operationId in a separate document; that stanza is the registration."""
lib = _lib()
spec = (
"paths:\n"
" /users/v1/{username}/password:\n"
" put:\n"
" security:\n"
" - bearerAuth: [ ]\n"
" operationId: api_views.users.update_password\n"
" /books/v1/{book_title}:\n"
" get:\n"
" security:\n"
" - bearerAuth: [ ]\n"
" summary: Only the owner may retrieve it\n"
" operationId: api_views.books.get_by_title\n"
)
stanza = lib["extract_registration_context"](spec, "get_by_title", "yaml")
assert "/books/v1/{book_title}:" in stanza
assert "operationId: api_views.books.get_by_title" in stanza
assert "security:" in stanza and "Only the owner may retrieve it" in stanza, "the guard the route declares is the point"
assert "update_password" not in stanza and "/users/v1" not in stanza
def test_registration_context_returns_a_whole_multiline_router_call() -> None:
"""An express router registers over several lines; the middleware between them is the guard."""
lib = _lib()
router = (
"webRouter.post(\n"
" '/project/:project_id/rename',\n"
" AuthorizationMiddleware.ensureUserCanAdminProject,\n"
" ProjectController.renameProject\n"
")\n"
"webRouter.post(\n"
" '/project/:project_id/export/:brand_variation_id',\n"
" AuthorizationMiddleware.ensureUserCanWriteProjectContent,\n"
" ExportsController.exportProject\n"
")\n"
"// exportProject is named in a comment too\n"
)
call = lib["extract_registration_context"](router, "exportProject", "javascript")
assert "ExportsController.exportProject" in call
assert "ensureUserCanWriteProjectContent" in call
assert "renameProject" not in call and "named in a comment" not in call
def test_registration_context_is_empty_when_the_document_never_names_the_handler() -> None:
lib = _lib()
assert lib["extract_registration_context"]("paths:\n /x:\n get: {}\n", "get_by_title", "yaml") == ""
# --- harvest -----------------------------------------------------------------
CONTEXT_RECIPE = {
"name": "vampi-books-get-by-title",
"repo": "erev0s/VAmPI",
"parent": "p" * 40,
"commit": "p" * 40,
"path": "api_views/books.py",
"relpath": "api_views/books.py",
"mode": "handler_context",
"function": "get_by_title",
"language": "python",
"license": "MIT",
"context": ["openapi_specs/openapi3.yml"],
}
HANDLER_SOURCE = "def get_by_title(book_title):\n return Book.query.filter_by(book_title=book_title).first()\n"
SPEC_SOURCE = (
"paths:\n"
" /books/v1/{book_title}:\n"
" get:\n"
" security:\n"
" - bearerAuth: [ ]\n"
" operationId: api_views.books.get_by_title\n"
)
def _patch_fetch(lib: dict[str, object], bodies: dict[str, str], fail: frozenset[str] = frozenset()) -> None:
"""`runpy.run_path` hands back a copy of the globals, so the module's own name has to be rebound."""
def fetch(url: str, timeout: int = 60) -> str:
for fragment in fail:
if fragment in url:
raise OSError(f"fetch failed: {url}")
for fragment, body in bodies.items():
if fragment in url:
return body
raise AssertionError(f"unexpected url {url}")
lib["harvest_recipe"].__globals__["fetch_url"] = fetch # type: ignore[attr-defined]
def test_context_documents_are_harvested_for_both_sides(tmp_path: Path) -> None:
lib = _lib()
_patch_fetch(lib, {"books.py": HANDLER_SOURCE, "openapi3.yml": SPEC_SOURCE})
lib["harvest_recipe"](CONTEXT_RECIPE, tmp_path)
for side in ("vuln", "fixed"):
doc = tmp_path / "vampi-books-get-by-title" / "context" / side / "openapi_specs" / "openapi3.yml"
assert doc.is_file(), f"{side} context document missing"
text = doc.read_text()
assert "# license: MIT" in text, "context documents carry the licence line too"
assert "operationId: api_views.books.get_by_title" in text
assert lib["context_paths"](CONTEXT_RECIPE, tmp_path) == (
(
"openapi_specs/openapi3.yml",
tmp_path / "vampi-books-get-by-title" / "context" / "vuln" / "openapi_specs" / "openapi3.yml",
tmp_path / "vampi-books-get-by-title" / "context" / "fixed" / "openapi_specs" / "openapi3.yml",
),
)
def test_context_documents_are_redacted_like_every_other_excerpt(tmp_path: Path) -> None:
lib = _lib()
secret = "ghp_" + "A" * 30
_patch_fetch(
lib,
{"books.py": HANDLER_SOURCE, "openapi3.yml": SPEC_SOURCE + f" description: token {secret}\n x: get_by_title\n"},
)
lib["harvest_recipe"](CONTEXT_RECIPE, tmp_path)
text = (tmp_path / "vampi-books-get-by-title" / "context" / "vuln" / "openapi_specs" / "openapi3.yml").read_text()
assert secret not in text and "***REDACTED***" in text
def test_a_failed_fetch_keeps_the_old_excerpt_and_records_the_reason(tmp_path: Path) -> None:
"""A re-harvest that cannot reach the network must not truncate a reviewed excerpt."""
lib = _lib()
folder = tmp_path / "vampi-books-get-by-title"
folder.mkdir(parents=True)
(folder / "vuln.py").write_text("# reviewed excerpt\n")
(folder / "fixed.py").write_text("# reviewed excerpt\n")
_patch_fetch(lib, {"books.py": HANDLER_SOURCE}, fail=frozenset({"openapi3.yml"}))
with pytest.raises(OSError):
lib["harvest_recipe"](CONTEXT_RECIPE, tmp_path)
assert (folder / "vuln.py").read_text() == "# reviewed excerpt\n", "a failed fetch rewrote a reviewed excerpt"
def test_the_driver_records_every_failed_fetch_with_its_reason(tmp_path: Path) -> None:
lib = _lib()
recipes = tmp_path / "recipes.toml"
recipes.write_text(
"[[recipe]]\n"
'name = "vampi-books-get-by-title"\n'
'repo = "erev0s/VAmPI"\n'
f'parent = "{"p" * 40}"\n'
f'commit = "{"p" * 40}"\n'
'path = "api_views/books.py"\n'
'relpath = "api_views/books.py"\n'
'mode = "handler_context"\n'
'function = "get_by_title"\n'
'language = "python"\n'
'license = "MIT"\n'
'context = ["openapi_specs/openapi3.yml"]\n'
)
_patch_fetch(lib, {"books.py": HANDLER_SOURCE}, fail=frozenset({"openapi3.yml"}))
report = tmp_path / "report.json"
code = lib["main"](["--recipes", str(recipes), "--out", str(tmp_path), "--all", "--fetch", "--force", "--report", str(report)])
assert code == 1
import json
rows = json.loads(report.read_text())
assert [row["name"] for row in rows] == ["vampi-books-get-by-title"]
assert rows[0]["status"] == "failed" and "openapi3.yml" in rows[0]["reason"]
# --- the entry-point mapper --------------------------------------------------
def _entry_point_names(root: Path) -> set[str]:
_, targets = preprocess_repository(root)
return {record.function_name for record in analyze_entry_points(root, targets) if record.function_name}
def test_an_openapi_operation_id_names_its_python_handler(tmp_path: Path) -> None:
"""Req 10.2: a handler registered from a spec document is an entry point, not an orphan function."""
(tmp_path / "api_views").mkdir()
(tmp_path / "api_views" / "books.py").write_text(HANDLER_SOURCE)
(tmp_path / "openapi_specs").mkdir()
(tmp_path / "openapi_specs" / "openapi3.yml").write_text(SPEC_SOURCE)
_, targets = preprocess_repository(tmp_path)
records = analyze_entry_points(tmp_path, targets)
named = [record for record in records if record.function_name == "get_by_title"]
assert named, "the operationId did not name its handler"
assert named[0].path == "api_views/books.py", "the record belongs to the handler file, not the spec"
assert named[0].access_level == "authenticated"
assert any("bearerAuth" in item or "security" in item for item in named[0].access_evidence)
def test_an_openapi_operation_without_security_is_public(tmp_path: Path) -> None:
(tmp_path / "api_views").mkdir()
(tmp_path / "api_views" / "books.py").write_text(HANDLER_SOURCE)
(tmp_path / "spec.yml").write_text("paths:\n /books:\n get:\n operationId: api_views.books.get_by_title\n")
assert "get_by_title" in _entry_point_names(tmp_path)
_, targets = preprocess_repository(tmp_path)
record = next(item for item in analyze_entry_points(tmp_path, targets) if item.function_name == "get_by_title")
assert record.access_level == "public"
def test_a_document_without_operation_ids_adds_no_entry_point(tmp_path: Path) -> None:
(tmp_path / "api_views").mkdir()
(tmp_path / "api_views" / "books.py").write_text(HANDLER_SOURCE)
(tmp_path / "compose.yml").write_text("services:\n web:\n image: python\n")
assert _entry_point_names(tmp_path) == set()
def test_es_and_commonjs_modules_are_javascript(tmp_path: Path) -> None:
"""`.mjs` was not a known suffix, so the one pair vendored as an ES module had no file target at all: no entry
point, no finding, and a pair that could only ever score as a miss."""
(tmp_path / "controller.mjs").write_text("export function handler(req, res) {}\n")
(tmp_path / "legacy.cjs").write_text("module.exports = function handler(req, res) {}\n")
_, targets = preprocess_repository(tmp_path)
assert {target.path: target.language for target in targets} == {"controller.mjs": "javascript", "legacy.cjs": "javascript"}
def test_the_overleaf_pair_is_scannable_at_all() -> None:
case = next(
item for item in select_vendored(load_pair_catalog(DEFAULT_CATALOG)) if item.name == "overleaf-with-claude-exportscontroller-3980b9"
)
with tempfile.TemporaryDirectory() as scratch:
root = Path(scratch)
target = root / case.relpath
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(case.vuln_file, target)
_, targets = preprocess_repository(root)
assert [item.language for item in targets] == ["javascript"]
def test_a_multiline_router_call_names_its_handler(tmp_path: Path) -> None:
(tmp_path / "Exports").mkdir()
(tmp_path / "Exports" / "ExportsController.mjs").write_text(
"async function exportProject(req, res, next) {\n return res.send(req.params.project_id)\n}\nexport default { exportProject }\n"
)
(tmp_path / "router.mjs").write_text(
"webRouter.post(\n"
" '/project/:project_id/export/:brand_variation_id',\n"
" AuthorizationMiddleware.ensureUserCanWriteProjectContent,\n"
" ExportsController.exportProject\n"
")\n"
)
_, targets = preprocess_repository(tmp_path)
records = [item for item in analyze_entry_points(tmp_path, targets) if item.function_name == "exportProject"]
assert records, "the multi-line registration named no handler"
assert any(record.access_level == "authenticated" for record in records)
# --- the corpus --------------------------------------------------------------
def _absence_rows() -> tuple[object, ...]:
taxonomy = load_families(FAMILIES)
rows = []
for case in select_vendored(load_pair_catalog(DEFAULT_CATALOG)):
for expected in case.expected:
declared = getattr(expected, "family", None)
family = declared or (
(taxonomy.family_of_cwe(expected.cwe or "") or taxonomy.family_of_mechanism(expected.mechanism or "")) or None
)
resolved = family if isinstance(family, str) else getattr(family, "id", "")
if resolved:
if resolved == "access_control":
rows.append(case)
break
return tuple(rows)
def test_every_absence_row_declares_its_family() -> None:
"""Req 1: the family is the key that routes the work; an absence row must not depend on a CWE lookup."""
rows = _absence_rows()
assert len(rows) >= 10
undeclared = [case.name for case in rows if not any(getattr(row, "family", None) for row in case.expected)]
assert undeclared == [], f"absence rows with no declared family: {undeclared}"
def test_absence_recipes_use_the_handler_context_mode() -> None:
names = {case.name for case in _absence_rows()}
seen = set()
for slice_name in ("vibe-py", "agent-vfc"):
for recipe in tomllib.loads((Path("benchmarks/pairs") / slice_name / "recipes.toml").read_text())["recipe"]:
if str(recipe["name"]) in names:
seen.add(str(recipe["name"]))
assert recipe.get("mode") == "handler_context", f"{recipe['name']} is not harvested with its handler context"
assert recipe.get("family") == "access_control", f"{recipe['name']} carries no family"
assert seen == names, f"absence rows with no recipe: {sorted(names - seen)}"
@pytest.mark.parametrize(
("name", "registration"),
[
("vampi-books-get-by-title", "operationId: api_views.books.get_by_title"),
("vampi-users-update-password", "operationId: api_views.users.update_password"),
("threatbyte-api-v1-get", "@ns.route('/profile')"),
("threatbyte-api-v1-delete", "@ns.route('/delete-user/<int:user_id>')"),
("openniw-jobs-0fc947", '@router.get("/{job_id}")'),
("overleaf-with-claude-exportscontroller-3980b9", "ExportsController.exportProject"),
],
)
def test_the_named_absence_rows_carry_their_registration(name: str, registration: str) -> None:
"""The observable of task 4.1: these excerpts used to arrive without the line that makes them reachable."""
case = next(item for item in select_vendored(load_pair_catalog(DEFAULT_CATALOG)) if item.name == name)
texts = [case.vuln_file.read_text(), case.fixed_file.read_text()]
texts += [vuln.read_text() for _, vuln, _ in case.context_files]
assert any(registration in text for text in texts), f"{name} lost its registration"
assert all("license:" in text for text in [case.vuln_file.read_text(), case.fixed_file.read_text()])
def test_every_absence_row_has_a_named_handler_or_a_stated_reason() -> None:
"""Either the mapper names the labeled handler, or the row says out loud why it cannot be scored."""
orphans = []
for case in _absence_rows():
with tempfile.TemporaryDirectory() as scratch:
root = Path(scratch) / "vuln"
root.mkdir()
target = root / case.relpath
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(case.vuln_file, target)
for relpath, vuln, _fixed in case.context_files:
context = root / relpath
context.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(vuln, context)
named = _entry_point_names(root)
labeled = {row.function for row in case.expected if row.function}
if not (labeled & named) and not case.unscorable:
orphans.append(f"{case.name} labeled={sorted(labeled)} named={sorted(named)}")
assert orphans == [], "absence rows with neither a named handler nor a stated reason:\n" + "\n".join(orphans)
def test_the_vendored_count_is_unchanged_by_the_re_harvest() -> None:
"""A re-harvest repairs excerpts; it must not quietly add or drop rows."""
cases = select_vendored(load_pair_catalog(DEFAULT_CATALOG))
per_slice: dict[str, int] = {}
for case in cases:
per_slice[case.slice] = per_slice.get(case.slice, 0) + 1
assert per_slice == {"agent-vfc": 29, "github": 6, "local": 3, "sast": 11, "vfc": 176, "vfc-js": 17, "vibe-py": 35}
def test_prose_inside_a_decorator_never_grants_a_guard(tmp_path: Path) -> None:
"""`@ns.doc(description='... without proper authorization checks')` describes the *bug*.
Matching auth words anywhere in a decorator, string literals included, reads an unguarded endpoint as guarded and
silences every obligation on it — the exact absence this corpus exists to catch."""
(tmp_path / "api.py").write_text(
"@ns.route('/delete-user/<int:user_id>')\n"
"@ns.doc(description='Delete a user without proper authorization checks. Broken Function Level Authorization.')\n"
"class UserDelete(Resource):\n"
" def delete(self, user_id):\n"
" return db.delete(user_id)\n\n"
"@ns.route('/settings')\n"
"class Settings(Resource):\n"
" @token_required\n"
" def get(self, current_user):\n"
" return current_user.settings\n"
)
_, targets = preprocess_repository(tmp_path)
access = {record.function_name: record.access_level for record in analyze_entry_points(tmp_path, targets) if record.function_name}
assert access == {"delete": "public", "get": "authenticated"}
tests/test_typescript_grammar.py
+3 -3
View File
@@ -46,6 +46,6 @@ def test_every_vendored_typescript_pair_parses_on_both_sides() -> None:
elif not ir.parse_ok:
failed.append(f"{case.name}:{side}")
assert unsupported == []
# One excerpt is a truncated destructuring parameter list, so it is not valid TypeScript on its own;
# that is a corpus defect for the re-harvest, not a grammar gap. Pinned exactly so a regression shows.
assert failed == ["lensops-kubernetesmanagement-fe4910:vuln", "lensops-kubernetesmanagement-fe4910:fixed"], failed
# Every vendored TypeScript excerpt now parses on both sides. The one that did not — a truncated destructuring
# parameter list — was a harvest defect, not a grammar gap, and task 4.1 re-harvested it as its own function.
assert failed == [], failed