mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 12:09:27 +00:00
measure: the candidate generator must be the IR, not the ruleset
A candidate-driven detector can only judge sites something else enumerates,
so the enumerator caps recall. Measured over the 58 scorable web pairs before
building anything:
pattern ruleset: 12.1% ceiling -- and the misses are not near-misses, the
ruleset finds nothing at all in most of these files
semantic IR: 96.6% ceiling, a median of ten call sites and bindings
inside the labeled function
The ruleset would have capped us at 12.1% against the 64.9% the unconstrained
loop already reaches. That design was one commit from being built.
Two corpus defects found on the way, both mine from task 4.1. The overleaf
excerpt started mid-import, because `_declarator_start` walks back to the
nearest `}` and Overleaf writes JavaScript without semicolons, so the nearest
one was inside `import { expressify }`. It then ended mid-object-literal,
because registration mentions were appended as bare lines. Neither parsed,
and a pair that cannot be parsed cannot be scored. Registration mentions now
expand to the statement that contains them; the ceiling moved 94.8% -> 96.6%.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
4e6afa862b
commit
309faad266
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"baseline_for_comparison": {
|
||||
"detection_rate": 0.649,
|
||||
"leak_rate": 0.123,
|
||||
"note": "deepseek-v4-flash, K=5, unconstrained tool loop, 57 scorable pairs across vibe-py and agent-vfc",
|
||||
"pair_correct": 0.526
|
||||
},
|
||||
"candidate_sources": {
|
||||
"pattern_ruleset (quick_scan_findings)": {
|
||||
"by_family": {
|
||||
"access_control": 0.0,
|
||||
"config_secrets": 0.0,
|
||||
"deserialization": 0.5,
|
||||
"injection": 0.27,
|
||||
"output_encoding": 0.0,
|
||||
"path": 0.0,
|
||||
"prototype": 0.0,
|
||||
"untrusted_destination": 0.0
|
||||
},
|
||||
"ceiling": 0.121,
|
||||
"note": "the ruleset finds nothing at all in most of these files, not merely nothing inside the labeled function",
|
||||
"pairs": 58,
|
||||
"pairs_with_a_candidate": 7
|
||||
},
|
||||
"semantic_ir (call sites + bindings)": {
|
||||
"ceiling": 0.966,
|
||||
"gaps": [
|
||||
"edutrace-page-ee7ba2 (labeled function absent from the IR)",
|
||||
"money-track-category-icons-4d4a5b (no calls or bindings in the labeled function)"
|
||||
],
|
||||
"pairs": 58,
|
||||
"pairs_with_a_candidate": 56,
|
||||
"sites_per_labeled_function": {
|
||||
"max": 33,
|
||||
"median": 10,
|
||||
"min": 1,
|
||||
"p90": 22
|
||||
}
|
||||
}
|
||||
},
|
||||
"command": "offline: parse each vulnerable side, count call sites and bindings inside the labeled function",
|
||||
"conclusion": "The candidate generator must be the semantic IR, not the pattern ruleset. The ruleset would cap recall at 12.1%, well below the 64.9% the unconstrained loop already reaches; the IR caps it at 96.6% and hands the model a median of ten sites to judge instead of a repository to search.",
|
||||
"generated": "2026-09-06",
|
||||
"note": "A candidate-driven detector can only judge sites something else enumerates, so the enumerator sets a hard ceiling on recall. Measured over the 58 scorable vendored web pairs, before building anything."
|
||||
}
|
||||
Reference in New Issue
Block a user