measure(model-grounded-detection): group 4 ceiling, 39.1% entailed against the flat IR's 2.2%

Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching
any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter
that makes no model calls at all.

Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36
distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10,
path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the
class no crash oracle can reach, arbitrated deterministically.

A hypothesis tested and discarded. From the group-2 misses I diagnosed path's
0/8 as a missing req.url source token and said so. The probe -- the same 8
pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real
cause is that the sink sits lexically inside an anonymous callback, so its
enclosing method is the closure (<lambda>0) and the sinkMethod == function
filter drops it; verdict(function="") on the same CPG returns model_entailed.

And the obvious fix for that is wrong. Relaxing the filter raises the number
with a FALSE witness: the flows found in the closure are `res -> readFileSync`
and `this -> ...`, neither of which is the vulnerability, whose real path
crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a
miss into a wrong entailment is worse than the miss -- it is the arbiter
asserting something untrue, the one failure mode this architecture exists to
prevent. The correct fix is two-part and recorded: scope a sink to any method
lexically nested in the labeled function, and scope parameter sources to that
function's own parameters, then re-measure counting only real witnesses.

Every miss is classified and none is architectural: 23 sink absent from the
excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source
unmodelled with the idiom named.

Two honesty items in the artifact. 16 of 92 pairs carry no function label and
are arbitrated over the whole file, so a verdict can credit a different finding
than the labeled one -- owasp-python-hash (CWE-327) is credited for a
permissive set_cookie elsewhere in its file, on both sides. And config_secrets
conflates three abstractions: permissive setting values (implemented), weak
algorithm choice (facts already carry weak_literals, unwired) and hardcoded
secrets (needs a credential-literal check).

694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-08 15:46:05 +02:00
co-authored by Claude Opus 5
parent eebca3f663
commit 91c0931a05
2 changed files with 1316 additions and 2 deletions
+61 -2
View File
@@ -191,7 +191,7 @@ queries in group 4). The gates, `redaction.py`, `pairs.py` overlay/inventory pat
## Group 4 — Absence and the rest of the taxonomy
- [ ] 4.1 Guard dominance for the absence families
- [x] 4.1 Guard dominance for the absence families
- `cpg/queries/dominance.sc` + `model/dominance.py::verdict(cpg, spec, candidate)` — `ENTAILED` when an
obligated operation is dominated by no discharger while its siblings are, seeded from the obligation
facts. This is the arbiter for the classes that never crash (access_control).
@@ -200,7 +200,7 @@ queries in group 4). The gates, `redaction.py`, `pairs.py` overlay/inventory pat
- _Requirements: 6.2, 7.3_
- _Depends: 3.2_
- [ ] 4.2 The remaining TaintSpecs and the ceiling regeneration
- [x] 4.2 The remaining TaintSpecs and the ceiling regeneration
- Fill `TaintSpec`s for path, deserialization, untrusted_destination, output_encoding, prototype, and the
config constant-abstraction arm (`model/taint.py`). Regenerate the full entailment ceiling per family and
per slice and commit it; name where each family falls back to `suspicion`.
@@ -414,3 +414,62 @@ Both sides arbitrated by the graph with the LLM never consulted — which is the
and the thing the prior architecture could not do at any price.
678 tests passing, mypy and ruff clean, three gates byte-identical, zero orphaned modules.
## Implementation Notes — group 4 (2026-09-08)
All three arbiter forms now work against a real CPG. Artifact:
`benchmarks/measurements/2026-09-08-model-entailment-ceiling-cpg.json`.
| | full model layer | flat IR baseline |
|---|---|---|
| `model_entailed` | **39.1%** (36/92) | 2.2% |
| any verdict | **45.6%** | 24% (corroborated, one-hop) |
| pairs distinguished | **23.9%** (22/92) | — |
Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33,
prototype 0.14, config_secrets 0.10, path 0.00. **access_control reaches 6 of 8 pairs and 0.38
distinguished** — the class no crash oracle can reach, arbitrated deterministically with no model call.
**4.1: "governed by a guard" is three relations, not one.** Measured against the engine rather than assumed:
an identity constraint lives *inside* the operation's arguments; an identity branch is a denial whose
`controlledBy` condition names the identity (`abort(403).controlledBy == [note.owner_id != current_user.id]`)
— it runs *after* the fetch, so it never dominates the operation but does control whether the value escapes;
only the third shape is plain dominance. That is why the query uses `controlledBy`.
**The bug that mattered most.** My first commit of 4.1 made the absence arbiter *silence its own family*:
`constraint_params` were treated as dischargers, so a field named `user_id` counted as a guard — but
`filter_by(id=user_id)` off the request is what an IDOR *is*. Compounded by substring matching, `user` matched
`users` inside `"SELECT * FROM users WHERE id = ?"`, and every operation in the handler read as guarded. On
`threatbyte-api-v1-delete` the arbiter returned no verdict at all on a textbook IDOR. Both fixed; the family
went from 0 verdicts to 6 of 8, every one distinguished.
**A hypothesis I tested and had to discard.** From the group-2 misses I diagnosed path's 0/8 as a missing
`req.url` source token, and said so. The probe — the same 8 pairs with and without the token — returned
**0/8 → 0/8**. Falsified. The real cause: the sink is lexically inside an anonymous callback, so its
enclosing method is the closure (`<lambda>0`) and the `sinkMethod == function` filter drops it;
`verdict(function="")` on the same CPG returns `model_entailed`.
**And the obvious fix for that is wrong.** Relaxing the filter raises the number with a *false witness*: the
flows found in the closure are `res -> fs.readFileSync(...)` and `this -> ...`, neither of which is the
vulnerability (the real path crosses a helper: `req.url -> url -> resolveUrl(...) -> possibleFilename`).
Turning a miss into a wrong entailment is worse than the miss — it is the arbiter asserting something untrue,
the one failure mode this architecture exists to prevent. The correct fix is two-part: scope a sink to any
method *lexically nested* in the labeled function, and scope parameter sources to the labeled function's own
parameters. Then re-measure, counting only witnesses that name a real source.
**Every miss is classified**, none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no
recognisable source in the excerpt, 3 source unmodelled (the idiom named).
**Two honesty items recorded in the artifact.** 16 of 92 pairs carry no function label and are therefore
arbitrated over the whole file, so a verdict can credit a different finding than the labeled one —
`owasp-python-hash` (CWE-327) is credited for a permissive `set_cookie` elsewhere in its file, on both sides.
And `config_secrets` (1/10) conflates three abstractions: permissive setting *values* (the arm implemented
here), weak *algorithm* choice (CWE-326/327 — the facts already carry `weak_literals` on the hashlib sink,
unwired), and hardcoded secrets (CWE-798, needing a credential-literal check); its settings table is also
Flask/FastAPI-shaped while the pairs are aiohttp. `hutool-cve-2018-17297` has no path spec at all — 4.2
authored path sinks for python and javascript only.
**Next, in evidence order:** the closure-scoping fix (with correct parameter scoping, then re-measure); Java
path sinks; the `weak_literals` arm for config; aiohttp settings.
693 tests passing, mypy and ruff clean, gates byte-identical, zero orphaned modules.
File diff suppressed because it is too large Load Diff