From 91c0931a050d5d112f38f65df426fddc389ec967 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Tue, 8 Sep 2026 15:46:05 +0200 Subject: [PATCH] measure(model-grounded-detection): group 4 ceiling, 39.1% entailed against the flat IR's 2.2% Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter that makes no model calls at all. Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10, path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the class no crash oracle can reach, arbitrated deterministically. A hypothesis tested and discarded. From the group-2 misses I diagnosed path's 0/8 as a missing req.url source token and said so. The probe -- the same 8 pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real cause is that the sink sits lexically inside an anonymous callback, so its enclosing method is the closure (0) and the sinkMethod == function filter drops it; verdict(function="") on the same CPG returns model_entailed. And the obvious fix for that is wrong. Relaxing the filter raises the number with a FALSE witness: the flows found in the closure are `res -> readFileSync` and `this -> ...`, neither of which is the vulnerability, whose real path crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a miss into a wrong entailment is worse than the miss -- it is the arbiter asserting something untrue, the one failure mode this architecture exists to prevent. The correct fix is two-part and recorded: scope a sink to any method lexically nested in the labeled function, and scope parameter sources to that function's own parameters, then re-measure counting only real witnesses. Every miss is classified and none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source unmodelled with the idiom named. Two honesty items in the artifact. 16 of 92 pairs carry no function label and are arbitrated over the whole file, so a verdict can credit a different finding than the labeled one -- owasp-python-hash (CWE-327) is credited for a permissive set_cookie elsewhere in its file, on both sides. And config_secrets conflates three abstractions: permissive setting values (implemented), weak algorithm choice (facts already carry weak_literals, unwired) and hardcoded secrets (needs a credential-literal check). 694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/model-grounded-detection/tasks.md | 63 +- ...26-09-08-model-entailment-ceiling-cpg.json | 1255 +++++++++++++++++ 2 files changed, 1316 insertions(+), 2 deletions(-) create mode 100644 benchmarks/measurements/2026-09-08-model-entailment-ceiling-cpg.json diff --git a/.kiro/specs/model-grounded-detection/tasks.md b/.kiro/specs/model-grounded-detection/tasks.md index 8aa25b8..475321b 100644 --- a/.kiro/specs/model-grounded-detection/tasks.md +++ b/.kiro/specs/model-grounded-detection/tasks.md @@ -191,7 +191,7 @@ queries in group 4). The gates, `redaction.py`, `pairs.py` overlay/inventory pat ## Group 4 — Absence and the rest of the taxonomy -- [ ] 4.1 Guard dominance for the absence families +- [x] 4.1 Guard dominance for the absence families - `cpg/queries/dominance.sc` + `model/dominance.py::verdict(cpg, spec, candidate)` — `ENTAILED` when an obligated operation is dominated by no discharger while its siblings are, seeded from the obligation facts. This is the arbiter for the classes that never crash (access_control). @@ -200,7 +200,7 @@ queries in group 4). The gates, `redaction.py`, `pairs.py` overlay/inventory pat - _Requirements: 6.2, 7.3_ - _Depends: 3.2_ -- [ ] 4.2 The remaining TaintSpecs and the ceiling regeneration +- [x] 4.2 The remaining TaintSpecs and the ceiling regeneration - Fill `TaintSpec`s for path, deserialization, untrusted_destination, output_encoding, prototype, and the config constant-abstraction arm (`model/taint.py`). Regenerate the full entailment ceiling per family and per slice and commit it; name where each family falls back to `suspicion`. @@ -414,3 +414,62 @@ Both sides arbitrated by the graph with the LLM never consulted — which is the and the thing the prior architecture could not do at any price. 678 tests passing, mypy and ruff clean, three gates byte-identical, zero orphaned modules. + +## Implementation Notes — group 4 (2026-09-08) + +All three arbiter forms now work against a real CPG. Artifact: +`benchmarks/measurements/2026-09-08-model-entailment-ceiling-cpg.json`. + +| | full model layer | flat IR baseline | +|---|---|---| +| `model_entailed` | **39.1%** (36/92) | 2.2% | +| any verdict | **45.6%** | 24% (corroborated, one-hop) | +| pairs distinguished | **23.9%** (22/92) | — | + +Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33, +prototype 0.14, config_secrets 0.10, path 0.00. **access_control reaches 6 of 8 pairs and 0.38 +distinguished** — the class no crash oracle can reach, arbitrated deterministically with no model call. + +**4.1: "governed by a guard" is three relations, not one.** Measured against the engine rather than assumed: +an identity constraint lives *inside* the operation's arguments; an identity branch is a denial whose +`controlledBy` condition names the identity (`abort(403).controlledBy == [note.owner_id != current_user.id]`) +— it runs *after* the fetch, so it never dominates the operation but does control whether the value escapes; +only the third shape is plain dominance. That is why the query uses `controlledBy`. + +**The bug that mattered most.** My first commit of 4.1 made the absence arbiter *silence its own family*: +`constraint_params` were treated as dischargers, so a field named `user_id` counted as a guard — but +`filter_by(id=user_id)` off the request is what an IDOR *is*. Compounded by substring matching, `user` matched +`users` inside `"SELECT * FROM users WHERE id = ?"`, and every operation in the handler read as guarded. On +`threatbyte-api-v1-delete` the arbiter returned no verdict at all on a textbook IDOR. Both fixed; the family +went from 0 verdicts to 6 of 8, every one distinguished. + +**A hypothesis I tested and had to discard.** From the group-2 misses I diagnosed path's 0/8 as a missing +`req.url` source token, and said so. The probe — the same 8 pairs with and without the token — returned +**0/8 → 0/8**. Falsified. The real cause: the sink is lexically inside an anonymous callback, so its +enclosing method is the closure (`0`) and the `sinkMethod == function` filter drops it; +`verdict(function="")` on the same CPG returns `model_entailed`. + +**And the obvious fix for that is wrong.** Relaxing the filter raises the number with a *false witness*: the +flows found in the closure are `res -> fs.readFileSync(...)` and `this -> ...`, neither of which is the +vulnerability (the real path crosses a helper: `req.url -> url -> resolveUrl(...) -> possibleFilename`). +Turning a miss into a wrong entailment is worse than the miss — it is the arbiter asserting something untrue, +the one failure mode this architecture exists to prevent. The correct fix is two-part: scope a sink to any +method *lexically nested* in the labeled function, and scope parameter sources to the labeled function's own +parameters. Then re-measure, counting only witnesses that name a real source. + +**Every miss is classified**, none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no +recognisable source in the excerpt, 3 source unmodelled (the idiom named). + +**Two honesty items recorded in the artifact.** 16 of 92 pairs carry no function label and are therefore +arbitrated over the whole file, so a verdict can credit a different finding than the labeled one — +`owasp-python-hash` (CWE-327) is credited for a permissive `set_cookie` elsewhere in its file, on both sides. +And `config_secrets` (1/10) conflates three abstractions: permissive setting *values* (the arm implemented +here), weak *algorithm* choice (CWE-326/327 — the facts already carry `weak_literals` on the hashlib sink, +unwired), and hardcoded secrets (CWE-798, needing a credential-literal check); its settings table is also +Flask/FastAPI-shaped while the pairs are aiohttp. `hutool-cve-2018-17297` has no path spec at all — 4.2 +authored path sinks for python and javascript only. + +**Next, in evidence order:** the closure-scoping fix (with correct parameter scoping, then re-measure); Java +path sinks; the `weak_literals` arm for config; aiohttp settings. + +693 tests passing, mypy and ruff clean, gates byte-identical, zero orphaned modules. diff --git a/benchmarks/measurements/2026-09-08-model-entailment-ceiling-cpg.json b/benchmarks/measurements/2026-09-08-model-entailment-ceiling-cpg.json new file mode 100644 index 0000000..0265c67 --- /dev/null +++ b/benchmarks/measurements/2026-09-08-model-entailment-ceiling-cpg.json @@ -0,0 +1,1255 @@ +{ + "flat_ir_baseline": { + "corroborated_one_hop": 0.239, + "entailed": 0.022, + "source": "2026-09-08-model-entailment-ceiling.json" + }, + "generated": "2026-09-08", + "java_gap": "hutool-cve-2018-17297 has no path spec at all: task 4.2 authored path sinks for python and javascript only.", + "miss_cause_note": "Classified statically from the excerpt text. `source_unmodelled` names a recognisable untrusted-input idiom the fact tables do not carry -- req.url (raw Node HTTP, what a static file server reads) and request.POST/GET (Django) are the two that recur. These are one-line data additions, not architecture.", + "miss_causes": { + "no_recognisable_source_in_excerpt": { + "examples": [ + "damn-vulnerable-graphql-application-views-resolve-pastes (injection)", + "djangoat-analytics-hits-by-ip (injection)", + "dsvpwa-handlers-do-bdr (injection)", + "dvpwa-student-create (injection)" + ], + "pairs": 13 + }, + "sink_absent_from_excerpt_or_unmodelled": { + "examples": [ + "sven-cwe-089-query-concat (injection)", + "hutool-cve-2018-17297 (path)", + "openssl-cve-2022-1292 (injection)", + "damn-vulnerable-graphql-application-views-mutate (injection)" + ], + "pairs": 23 + }, + "source_unmodelled": { + "examples": [ + "djangoat-views-reset-password (deserialization, request.POST)", + "dsvpwa-attacks-run (injection, handler.params)", + "angular-http-server-1-0-0 (path, req.url)" + ], + "pairs": 3 + } + }, + "overall": { + "any_verdict_rate": 0.4565, + "corroborated": 6, + "distinguished": 22, + "distinguished_rate": 0.2391, + "entailed": 36, + "entailed_rate": 0.3913, + "none": 50, + "pairs": 92 + }, + "path_family_probe": { + "actual_cause": "The sink is lexically inside an anonymous callback, so its enclosing method is the closure rather than the labeled function, and the verdict's `sinkMethod == function` filter drops it. On angular-http-server the sink `fs.readFileSync(possibleFilename)` sits in the callback passed to fs.stat, and Joern names its method `0`; verdict(function='requestListener') returns None while verdict(function='') returns model_entailed. JavaScript is callback-heavy, so the labeled function and the method containing the sink routinely differ.", + "correct_fix": [ + "scope a sink to any method lexically nested within the labeled function, not only the labeled function itself", + "scope parameter sources to the labeled function's own parameters, not every enclosing or lambda parameter", + "then re-measure: only a witness naming the real source should count" + ], + "method": "the same 8 path pairs, run with and without req.url/request.url/req.path/url.parse added to the source facts", + "question": "Does the missing `req.url` source token explain path's 0/8?", + "result": { + "with_req_url_added": "0/8", + "with_shipped_facts": "0/8" + }, + "verdict": "FALSIFIED. The source token was not the cause, and adding it converts nothing.", + "why_the_obvious_fix_is_wrong": "Relaxing the filter alone raises the number with a FALSE witness. With parameter sources on, the flows found in that closure are `res -> fs.readFileSync(...)` and `this -> ...`, neither of which is the vulnerability -- the real path is req.url -> url -> resolveUrl(...) -> possibleFilename -> sink, which also crosses a helper call. Turning a miss into a wrong entailment is worse than the miss: it is the arbiter asserting something untrue, which is the one failure mode this architecture exists to prevent." + }, + "per_family": { + "access_control": { + "any_verdict_rate": 0.75, + "corroborated": 6, + "distinguished": 3, + "distinguished_rate": 0.375, + "entailed": 0, + "entailed_rate": 0.0, + "none": 2, + "pairs": 8 + }, + "config_secrets": { + "any_verdict_rate": 0.1, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 1, + "entailed_rate": 0.1, + "none": 9, + "pairs": 10 + }, + "deserialization": { + "any_verdict_rate": 0.3333, + "corroborated": 0, + "distinguished": 1, + "distinguished_rate": 0.3333, + "entailed": 1, + "entailed_rate": 0.3333, + "none": 2, + "pairs": 3 + }, + "injection": { + "any_verdict_rate": 0.58, + "corroborated": 0, + "distinguished": 18, + "distinguished_rate": 0.36, + "entailed": 29, + "entailed_rate": 0.58, + "none": 21, + "pairs": 50 + }, + "output_encoding": { + "any_verdict_rate": 0.0, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 0, + "entailed_rate": 0.0, + "none": 1, + "pairs": 1 + }, + "path": { + "any_verdict_rate": 0.0, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 0, + "entailed_rate": 0.0, + "none": 8, + "pairs": 8 + }, + "prototype": { + "any_verdict_rate": 0.1429, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 1, + "entailed_rate": 0.1429, + "none": 6, + "pairs": 7 + }, + "untrusted_destination": { + "any_verdict_rate": 0.8, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 4, + "entailed_rate": 0.8, + "none": 1, + "pairs": 5 + } + }, + "per_slice": { + "agent-vfc": { + "any_verdict_rate": 0.2857, + "corroborated": 3, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 5, + "entailed_rate": 0.1786, + "none": 20, + "pairs": 28 + }, + "github": { + "any_verdict_rate": 0.6667, + "corroborated": 0, + "distinguished": 2, + "distinguished_rate": 0.3333, + "entailed": 4, + "entailed_rate": 0.6667, + "none": 2, + "pairs": 6 + }, + "local": { + "any_verdict_rate": 1.0, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 2, + "entailed_rate": 1.0, + "none": 0, + "pairs": 2 + }, + "sast": { + "any_verdict_rate": 1.0, + "corroborated": 0, + "distinguished": 4, + "distinguished_rate": 0.5, + "entailed": 8, + "entailed_rate": 1.0, + "none": 0, + "pairs": 8 + }, + "vfc": { + "any_verdict_rate": 0.0, + "corroborated": 0, + "distinguished": 0, + "distinguished_rate": 0.0, + "entailed": 0, + "entailed_rate": 0.0, + "none": 1, + "pairs": 1 + }, + "vfc-js": { + "any_verdict_rate": 0.4706, + "corroborated": 0, + "distinguished": 4, + "distinguished_rate": 0.2353, + "entailed": 8, + "entailed_rate": 0.4706, + "none": 9, + "pairs": 17 + }, + "vibe-py": { + "any_verdict_rate": 0.4, + "corroborated": 3, + "distinguished": 12, + "distinguished_rate": 0.4, + "entailed": 9, + "entailed_rate": 0.3, + "none": 18, + "pairs": 30 + } + }, + "remeasured": { + "families": [ + "access_control", + "config_secrets" + ], + "pairs": 18, + "why": "access_control was measured before the constraint_params / word-boundary fixes, which had it reporting every operation as guarded; config_secrets was routed through taint instead of the constant-abstraction arm. The flow families were unaffected and keep their original numbers." + }, + "rows": [ + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "python", + "pair": "local-python-vulnerable", + "seconds": 32.4, + "slice": "local", + "vuln_rung": "model_entailed", + "witness": "request.args[\"expr\"] -> eval(request.args[\"expr\"]) (line 28, 1 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "local-javascript-vulnerable", + "seconds": 30.5, + "slice": "local", + "vuln_rung": "model_entailed", + "witness": "\"make \" + req.query.target -> childProcess.execSync(\"make \" + req.query.target) (line 14, 1 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "python", + "pair": "sven-cwe-078-os-system", + "seconds": 26.9, + "slice": "github", + "vuln_rung": "model_entailed", + "witness": "username -> os.system(\"useradd -G docker,wheel -p \" + encPass + \" \" + username) (line 10, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "python", + "pair": "sven-cwe-078-shell-true", + "seconds": 28.6, + "slice": "github", + "vuln_rung": "model_entailed", + "witness": "cmd -> _exec_cmd(self, cmd) (line 14, 2 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "sven-cwe-089-execute-percent", + "seconds": 27.9, + "slice": "github", + "vuln_rung": "model_entailed", + "witness": "game -> cursor.execute(\"SELECT x,y,letter FROM moves WHERE gameID = %d;\" % game) (line 12, 3 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "sven-cwe-089-query-concat", + "seconds": 27.1, + "slice": "github", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "java", + "pair": "hutool-cve-2018-17297", + "seconds": 29.4, + "slice": "github", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "node-serialize-cve-2017-5941", + "seconds": 28.2, + "slice": "github", + "vuln_rung": "model_entailed", + "witness": "obj -> eval(\"(\" + obj + \")\") (line 4, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "python", + "pair": "owasp-python-codeinj", + "seconds": 28.0, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "urllib.parse.unquote_plus(request.cookies.get(\"BenchmarkTest00074\", \"noCookieValueSupplied\")) -> exec(bar) (line 51, 9 steps)" + }, + { + "distinguished": true, + "family": "deserialization", + "fixed_rung": "none", + "language": "python", + "pair": "owasp-python-deser", + "seconds": 27.6, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "urllib.parse.unquote_plus(request.cookies.get(\"BenchmarkTest00078\", \"noCookieValueSupplied\")) -> pickle.loads(base64.urlsafe_b64decode(bar)) (line 53, 12 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "owasp-python-cmdi", + "seconds": 27.7, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "request.form.get(\"BenchmarkTest00165\") -> subprocess.run(argList, capture_output = True, encoding = \"utf-8\") (line 57, 15 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "owasp-python-sqli", + "seconds": 28.0, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "request.form.get(\"BenchmarkTest00099\") -> cur.execute(sql) (line 49, 14 steps)" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "model_entailed", + "language": "python", + "pair": "owasp-python-hash", + "seconds": 30.2, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "response.set_cookie('BenchmarkTest00054', 'someSecret', max_age = 60 * 3, secure = True, path = request.path, domain = 'localhost') (line 26): permissive literal True" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "java", + "pair": "owasp-java-cmdi", + "seconds": 49.7, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "getHeader(\"BenchmarkTest00007\") -> r.exec(args, argsEnv) (line 61, 10 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "java", + "pair": "owasp-java-sqli", + "seconds": 48.8, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "getHeader(\"BenchmarkTest00008\") -> connection.prepareCall(sql) (line 57, 9 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "java", + "pair": "juliet-java-cwe78-exec", + "seconds": 47.9, + "slice": "sast", + "vuln_rung": "model_entailed", + "witness": "String data -> Runtime.getRuntime().exec(osCommand + data) (line 8, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "openssl-cve-2022-1292", + "seconds": 1.2, + "slice": "vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "damn-vulnerable-graphql-application-views-mutate", + "seconds": 26.8, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "damn-vulnerable-graphql-application-views-resolve-pastes", + "seconds": 26.9, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "damn-vulnerable-graphql-application-views-resolve-system-dia", + "seconds": 26.8, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "djangoat-analytics-hits-by-ip", + "seconds": 30.4, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "deserialization", + "fixed_rung": "none", + "language": "python", + "pair": "djangoat-views-reset-password", + "seconds": 30.0, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "djangoat-views-user-view", + "seconds": 31.2, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "request -> User.objects.raw(\"SELECT * FROM %s WHERE user_id='%s' ORDER BY is_admin='0'\" % (table_name,user_id_form)) (line 22, 9 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "dsvpwa-attacks-run", + "seconds": 52.3, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "python", + "pair": "dsvpwa-dsvpwa-main", + "seconds": 30.5, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "dsvpwa-handlers-do-bdr", + "seconds": 41.2, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "python", + "pair": "dvpwa-app-init", + "seconds": 29.0, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "python", + "pair": "dvpwa-middlewares-session-middleware", + "seconds": 28.7, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "dvpwa-student-create", + "seconds": 40.0, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "deserialization", + "fixed_rung": "none", + "language": "python", + "pair": "extremely-vulnerable-flask-app-account-before-request", + "seconds": 30.8, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "extremely-vulnerable-flask-app-account-search", + "seconds": 28.0, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "request.args.get('search', '') -> text(f\"text like '%{search_param}%'\") (line 20, 6 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "extremely-vulnerable-flask-app-signup-validate-token", + "seconds": 27.7, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "session -> session.execute(text(f\"\"\"\n SELECT id, code FROM {RegistrationCode.__tablename__} WHERE code = '{code}'\n \"\"\")) (line 15, 2 steps)" + }, + { + "distinguished": true, + "family": "access_control", + "fixed_rung": "none", + "language": "python", + "pair": "threatbyte-api-v1-delete", + "seconds": 27.7, + "slice": "vibe-py", + "vuln_rung": "model_corroborated", + "witness": "delete line 28: obligated operation with no discharging guard, and no guarded sibling" + }, + { + "distinguished": true, + "family": "access_control", + "fixed_rung": "none", + "language": "python", + "pair": "threatbyte-api-v1-get", + "seconds": 30.8, + "slice": "vibe-py", + "vuln_rung": "model_corroborated", + "witness": "get line 27: obligated operation with no discharging guard, and no guarded sibling" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "threatbyte-routes-dashboard", + "seconds": 27.8, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "request.args.get('search', '') -> cursor.execute(s_query) (line 29, 8 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vampi-user-model-get-user", + "seconds": 27.4, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "access_control", + "fixed_rung": "none", + "language": "python", + "pair": "vampi-users-update-password", + "seconds": 30.3, + "slice": "vibe-py", + "vuln_rung": "model_corroborated", + "witness": "update_password line 16: obligated operation with no discharging guard, and no guarded sibling" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vfapi-main-delete-user", + "seconds": 27.6, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vfapi-main-put-user", + "seconds": 27.0, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vfapi-main-sql-return-users-from-username", + "seconds": 26.7, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "output_encoding", + "fixed_rung": "none", + "language": "python", + "pair": "vulnerable-python-apps-bad-mvc-do-get", + "seconds": 30.2, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "python", + "pair": "vulnerable-python-apps-dir-traversal-do-get", + "seconds": 38.2, + "slice": "vibe-py", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vulnpy-cmdi-do-os-system", + "seconds": 45.5, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "command -> os.system(command) (line 14, 2 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vulnpy-cmdi-do-subprocess-popen", + "seconds": 31.3, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "command -> subprocess.Popen(command, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, shell = True) (line 14, 2 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vulpy-libuser-create", + "seconds": 35.7, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "password -> c.execute(\"INSERT INTO users (username, password, failures, mfa_enabled, mfa_secret) VALUES ('%s', '%s', '%d', '%d', '%s')\" % (username,password,0,0,'')) (line 18, 4 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vulpy-libuser-login", + "seconds": 54.1, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "username -> c.execute(\"SELECT * FROM users WHERE username = '{}' and password = '{}'\".format(username, password)) (line 20, 5 steps)" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "vulpy-libuser-password-change", + "seconds": 49.6, + "slice": "vibe-py", + "vuln_rung": "model_entailed", + "witness": "password -> c.execute(\"UPDATE users SET password = '{}' WHERE username = '{}'\".format(password, username)) (line 20, 5 steps)" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "javascript", + "pair": "angular-http-server-1-0-0", + "seconds": 32.3, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "apex-publish-static-files-2-0-0", + "seconds": 32.2, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "this -> execSync(\n\t\t\t\t'\"' + opts.sqlclPath + '\"' + // Sqlcl path\n\t\t\t\t' ' + opts.connectString + // Connect string (user/pass@server:port/sid)\n\t\t\t\t' @\"' + path.resolve(__dirname, 'lib/script') + '\"' + // Sql t (line 61, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "bestzip-2-1-6", + "seconds": 29.1, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "clamscan-1-2-0", + "seconds": 46.0, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "this -> execFile(self.settings[self.scanner].path, self._build_clam_args(path), (err, stdout, stderr) => {\n const {is_infected, viruses} = self._process_result(stdout, path);\n\n (line 1401, 49 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "diskusage-ng-0-2-6", + "seconds": 31.5, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "this -> exec('df -k \"' + path + '\"', function(err, stdout) {\n if (err) {\n return cb(err);\n }\n\n try {\n cb(null, parse(stdout));\n } catch (e) {\n cb(e (line 19, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "dns-sync-0-1-0", + "seconds": 30.6, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "hostname -> shell.exec(cmd, {silent: true}) (line 22, 7 steps)" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "javascript", + "pair": "glance-3-0-0", + "seconds": 29.5, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "hot-formula-parser-3-0-0", + "seconds": 47.8, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "yytext -> eval(\"[\" + yytext + \"]\") (line 166, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "libnmap-0-4-11", + "seconds": 31.1, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "pdf-image-1-0-5", + "seconds": 30.9, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "pidusage-1-0-0", + "seconds": 31.6, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "this -> exec(cmd + pid, function(err, stdout, stderr) {\n if(err)\n return done(err, null)\n\n stdout = stdout.split(os.EOL)[1]\n stdout = stdout.replace(/^\\s+/, '').replace(/\\s\\s+/g, ' ').sp (line 116, 2 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "portprocesses-1-0-4", + "seconds": 29.0, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "javascript", + "pair": "serve-here-js-1-1-0", + "seconds": 28.6, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "static-eval-1-1-1", + "seconds": 35.5, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "thenify-3-3-0", + "seconds": 31.5, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "this -> eval(createWrapper($$__fn__$$.name, options)) (line 29, 4 steps)" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "underscore-1-13-0-0", + "seconds": 30.2, + "slice": "vfc-js", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": true, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "whereis-0-4-0", + "seconds": 29.1, + "slice": "vfc-js", + "vuln_rung": "model_entailed", + "witness": "name -> cp.exec('which ' + name, function(error, stdout, stderr) {\n stdout = stdout.split('\\n')[0];\n if (error || stderr || stdout === '' || stdout.charAt(0) !== '/') {\n stdout = stdout.split('\\n') (line 15, 3 steps)" + }, + { + "distinguished": false, + "family": "untrusted_destination", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "anastasios-pw-session-14ec1a", + "seconds": 28.6, + "slice": "agent-vfc", + "vuln_rung": "model_entailed", + "witness": "request: Request -> request.isNavigationRequest (line 14, 2 steps)" + }, + { + "distinguished": false, + "family": "untrusted_destination", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "astroclaw-pw-session-1553a8", + "seconds": 28.7, + "slice": "agent-vfc", + "vuln_rung": "model_entailed", + "witness": "request: Request -> request.isNavigationRequest (line 14, 2 steps)" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "javascript", + "pair": "capgo-app-hono-4dadd2", + "seconds": 30.7, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "python", + "pair": "dnf-plugin-anyrepo-config-26d19a", + "seconds": 27.4, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "edutrace-page-ee7ba2", + "seconds": 33.8, + "slice": "agent-vfc", + "vuln_rung": "model_entailed", + "witness": "this -> setFormData(prevData => {\n const newData = JSON.parse(JSON.stringify(prevData));\n let current = newData;\n for (let i = 0; i < keys.length - 1; i++) {\n current = current[keys[i]]; (line 65, 2 steps)" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "none", + "language": "javascript", + "pair": "etude-lexical-markdown-editorconfig-c61a8b", + "seconds": 28.7, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "none", + "language": "javascript", + "pair": "etude-lexical-markdown-editorconfig-db4f54", + "seconds": 28.9, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "untrusted_destination", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "market-cap-route-4f8ae9", + "seconds": 29.6, + "slice": "agent-vfc", + "vuln_rung": "model_entailed", + "witness": "request: NextRequest -> request.url (line 14, 2 steps)" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "javascript", + "pair": "melodix-playerprovider-089931", + "seconds": 30.1, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "javascript", + "pair": "mercado-do-vale-news-catalogmetadataservice-9b5fcf", + "seconds": 28.5, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "none", + "language": "javascript", + "pair": "money-track-category-icons-4d4a5b", + "seconds": 28.9, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "none", + "language": "javascript", + "pair": "nestjs-filter-normalizer-aa0e60", + "seconds": 29.2, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "javascript", + "pair": "new-erp-final-app-6e9644", + "seconds": 30.7, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "untrusted_destination", + "fixed_rung": "model_entailed", + "language": "javascript", + "pair": "nimestream-route-d17f53", + "seconds": 30.5, + "slice": "agent-vfc", + "vuln_rung": "model_entailed", + "witness": "request -> request.url (line 14, 2 steps)" + }, + { + "distinguished": false, + "family": "access_control", + "fixed_rung": "model_corroborated", + "language": "python", + "pair": "openniw-jobs-0fc947", + "seconds": 26.8, + "slice": "agent-vfc", + "vuln_rung": "model_corroborated", + "witness": "get_job line 16: obligated operation with no discharging guard, and no guarded sibling" + }, + { + "distinguished": false, + "family": "access_control", + "fixed_rung": "model_corroborated", + "language": "python", + "pair": "openniw-jobs-b74de3", + "seconds": 26.9, + "slice": "agent-vfc", + "vuln_rung": "model_corroborated", + "witness": "get_job line 16: obligated operation with no discharging guard, and no guarded sibling" + }, + { + "distinguished": false, + "family": "access_control", + "fixed_rung": "none", + "language": "javascript", + "pair": "overleaf-with-claude-exportscontroller-3980b9", + "seconds": 1.1, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "none", + "language": "javascript", + "pair": "popmark-editorconfig-e5ac40", + "seconds": 32.1, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "prototype", + "fixed_rung": "none", + "language": "javascript", + "pair": "rag-pipeline-utils-config-handler-76e5f0", + "seconds": 31.8, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "javascript", + "pair": "registry-proxy-worker-426847", + "seconds": 28.4, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "access_control", + "fixed_rung": "model_corroborated", + "language": "javascript", + "pair": "supplieradvisor-mvp-route-387bcf", + "seconds": 31.8, + "slice": "agent-vfc", + "vuln_rung": "model_corroborated", + "witness": "PATCH line 37: obligated operation with no discharging guard, and no guarded sibling" + }, + { + "distinguished": false, + "family": "access_control", + "fixed_rung": "none", + "language": "javascript", + "pair": "supplieradvisor-mvp-route-a69e8f", + "seconds": 30.4, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "python", + "pair": "tools-cli-tools-a96bc0", + "seconds": 29.0, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "javascript", + "pair": "trackly-cli-path-diagnostics-1c1aed", + "seconds": 29.9, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "path", + "fixed_rung": "none", + "language": "python", + "pair": "upstreamdrift-local-server-c3a078", + "seconds": 28.4, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "untrusted_destination", + "fixed_rung": "none", + "language": "python", + "pair": "waggledance-swarm-ssrf-host-guard-9a4548", + "seconds": 29.3, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "injection", + "fixed_rung": "none", + "language": "javascript", + "pair": "workers-research-utils-209444", + "seconds": 31.8, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + }, + { + "distinguished": false, + "family": "config_secrets", + "fixed_rung": "none", + "language": "javascript", + "pair": "zoonk-origin-bfbf67", + "seconds": 27.8, + "slice": "agent-vfc", + "vuln_rung": "none", + "witness": "" + } + ], + "scope_caveat": { + "entailed_among_them": 14, + "examples": [ + "hutool-cve-2018-17297", + "juliet-java-cwe78-exec", + "local-javascript-vulnerable", + "local-python-vulnerable", + "node-serialize-cve-2017-5941", + "owasp-java-cmdi" + ], + "note": "These are arbitrated over the whole file rather than a labeled function, so a verdict may be about a real finding other than the labeled one. owasp-python-hash is the clear case: a CWE-327 weak-hash pair credited for a permissive set_cookie elsewhere in the file, on both sides.", + "of_total": 92, + "pairs_without_a_function_label": 16 + }, + "task": "model-grounded-detection 4.2 \u2014 entailment ceiling with the full model layer (taint + dominance)" +}