marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

measure: final push-hook arm on all 252 pilot changes (0/12, 2.1 false alarms per 100) and the combined whole-tool score

Browse Source 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-10-13 15:50:29 +02:00
co-authored by Claude Opus 5.5
parent ec40dc1c93
commit a99579b547
5 changed files with 674 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-10-12-valid-benchmark-pilot/NOTES.md
+19 -19
View File
@@ -10,28 +10,27 @@ Two arms over the same 252 changes (12 verified positives, 240 ordinary pushes f
repos), as R6 of the requirements asks:
- **(a) push hook as shipped**: quick rules + engine, 300 s deadline, the Docker replay lane, no
model key in the container. Still running when this was written: 163 of 252 records done (all 12
positives, 151 ordinary). The final numbers land in `MORNING.txt` outside the repo and will replace
the `*_interim_163.json` files here.
model key in the container. Finished 2026-10-13 15:38: 252 of 252 records (12 positives, 240
ordinary).
- **(b) hunter arm**: the LLM tool hunter (`scan --mode deep`'s search) run off-cluster over the
changed files at head, every finding classified into the taxonomy by the decision-engine
classifier. Finished: 252 of 252. Reported twice: raw (every hunter suspicion counts) and
filtered (only suspicions the classifier judged `vulnerable`).
| Metric | (a) push hook, 163 done | (b) hunter, raw | (b') hunter, classifier-filtered |
|---|---|---|---|
| Recall (12 positives) | 0 / 12 = 0% (Wilson 0 to 24.3) | 6 / 12 = 50% (25.4 to 74.6) | 5 / 12 = 41.7% (19.3 to 68.0) |
| False alarms per 100 ordinary pushes | 1 / 149 = 0.7 (0.1 to 3.7) | 221 / 239 = 92.5 (88.4 to 95.2) | 130 / 239 = 54.4 (48.1 to 60.6) |
| Precision (flagged pushes that were positives) | 0 / 1 | 6 / 232 = 2.6% | 5 / 139 = 3.6% |
| Exploit-verified | 0 (no proof run) | 0 (no proof run) | 0 (no proof run) |
| Instrument failures | 2 / 163 | 1 / 252 | 1 / 252 |
| Metric | (a) push hook | (b) hunter, raw | (b') hunter, classifier-filtered | (a) + (b') whole tool |
|---|---|---|---|---|
| Recall (12 positives) | 0 / 12 = 0% (Wilson 0 to 24.3) | 6 / 12 = 50% (25.4 to 74.6) | 5 / 12 = 41.7% (19.3 to 68.0) | 5 / 12 = 41.7% (19.3 to 68.0) |
| False alarms per 100 ordinary pushes | 5 / 238 = 2.1 (0.9 to 4.8) | 221 / 239 = 92.5 (88.4 to 95.2) | 130 / 239 = 54.4 (48.1 to 60.6) | 132 / 238 = 55.5 (49.1 to 61.6) |
| Precision (flagged pushes that were positives) | 0 / 5 | 6 / 232 = 2.6% | 5 / 139 = 3.6% | 5 / 141 = 3.5% |
| Exploit-verified | 0 (no proof run) | 0 (no proof run) | 0 (no proof run) | 0 (no proof run) |
| Instrument failures | 2 / 252 | 1 / 252 | 1 / 252 | 2 / 252 |
Neither arm meets the gate (recall at least 90%, false alarms under 10 per 100 pushes). The hook as
shipped is quiet and blind on this set. The hunter finds about half and alarms on more than half of
ordinary pushes even after the classifier. The whole-tool number the brief holds the tool to is
(a) plus (b'); since (a) caught nothing, it equals (b') on recall and adds one false alarm.
(a) plus (b'): 41.7% recall at 55.5 false alarms per 100 pushes. (a) caught nothing, so recall equals (b'); of its five false alarms, two were on pushes (b') had already flagged and three were new. The denominator is 238, not 239, because (a) has two instrument failures among the ordinary pushes.
Files: `scored_a_interim_163.json`, `metrics_a_interim_163.json`, `scored_b_raw.json`,
Files: `scored_a.json`, `metrics_a.json`, `scored_combined.json`, `metrics_combined.json`, `scored_b_raw.json`,
`metrics_b_raw.json`, `scored_b_filtered.json`, `metrics_b_filtered.json`. Scorer:
`benchmarks/unseen/score.py` (a catch needs the labelled family and a location inside the labelled
function or within 10 lines of it). Metrics: `benchmarks/valid/metrics.py`.
@@ -59,13 +58,16 @@ strict scorer; note them.
### Arm (a) detail
- Engine finished inside 300 s on 100 of 163 changes (61%); 61 fell back to quick rules only; 2
instrument failures (`contentful` 9e706114, `matrix` 7018468e, nothing readable at the hotspots).
On the positives: engine ran on 4 of 12, quick only on 8.
- Engine finished inside 300 s on 116 of 252 changes (46%); 134 fell back to quick rules only; 2
instrument failures (`contentful` d5dac236, push artifact unread; `matrix` fed2acfb, no changed
bytes read). On the positives: engine ran on 4 of 12, quick only on 8.
- Every record proves the instrument: hook exit 0, bytes read per changed file, base and head
verified.
- The one false alarm: `bytebase/dbhub` 5bd45460, quick rule `js-command-exec` (CWE-78) at
`src/connectors/sqlite/index.ts:132`, "Node child_process command execution needs review".
- All five false alarms are quick rules; the engine raised none:
- `bytebase/dbhub` c10e1269, `js-command-exec` (CWE-78) at `src/connectors/sqlite/index.ts:132`
- `nvbn/thefuck` 6e325eb7, `python-os-command` (CWE-78) at `thefuck/entrypoints/shell_logger.py:35`
- `zenitram/malla` dea8d513 and 61c4fd5e, `python-sql-injection` (CWE-89) in the repository and analytics modules
- `zereight/gitlab-mcp` 3d01ea54, `js-command-exec` (CWE-78) at `scripts/generate-tools-readme.ts:28`
### Arm (b) detail
@@ -172,8 +174,6 @@ change.
## Not done yet
- Final arm (a) numbers on all 252 changes (replay finishing 2026-10-13 afternoon), then the
combined (a)+(b') score with `hunter_arm combine --keep-verdicts vulnerable`.
- Exploit-verified rate: the proof arm has not run on any of the 26 flagged positive findings.
- Growing from 12 to the ~40 positives `score.py:required_n` asks for.
- The 252-change replay at the engine's ~200 s fixed cost is about 17 hours one container at a time
benchmarks/measurements/2026-10-12-valid-benchmark-pilot/metrics_a_interim_163.json → benchmarks/measurements/2026-10-12-valid-benchmark-pilot/metrics_a.json
+11 -11
View File
@@ -14,26 +14,26 @@
},
"false_alarms_per_100_pushes": {
"cluster95": [
0.0,
2.027027027027027
0.42194092827004215,
4.166666666666666
],
"count": 1,
"icc": -0.004445973772031825,
"n": 149,
"rate": 0.6711409395973155,
"count": 5,
"icc": 0.004133404058911234,
"n": 238,
"rate": 2.100840336134454,
"wilson95": [
0.11856820977550406,
3.703427895996746
0.9005985705189141,
4.822817086671577
]
},
"headline": "On unseen app repos: recall 0.0% (Wilson 0.0-24.3), false alarms 0.7 per 100 pushes, precision 0.0%; exploit-verified 0 of 0.",
"headline": "On unseen app repos: recall 0.0% (Wilson 0.0-24.3), false alarms 2.1 per 100 pushes, precision 0.0%; exploit-verified 0 of 0.",
"precision": {
"count": 0,
"n": 1,
"n": 5,
"rate": 0.0,
"wilson95": [
0,
0.7934567085261071
0.43449149475208104
]
},
"recall": {
benchmarks/measurements/2026-10-12-valid-benchmark-pilot/metrics_combined.json
+79
View File
@@ -0,0 +1,79 @@
{
"exploit_verified": {
"count": 0,
"flagged_positive_pushes": 9,
"flagged_positives": 26,
"n": 0,
"not_yet_proved": [
"contentful-contentful-mcp-server-22e8b2fc09c3b2c4",
"contentful-contentful-mcp-server-df53a40a29ee8d8d",
"i18next-i18next-http-middleware-f3e2a6886951f36a",
"imzbf-md-editor-v3-14658213915dbde5",
"imzbf-md-editor-v3-b9c833a3524a57a9",
"jupyter-lsp-jupyterlab-lsp-59f9a8b80b133f6f",
"nvbn-thefuck-456ad18acb14f1b2",
"nvbn-thefuck-74126de049d52d8c",
"ondata-ckan-mcp-server-01a3f68f84397eb4",
"ondata-ckan-mcp-server-b0b05cdc830b1f8b",
"webpack-webpack-bundle-analyzer-8e07af8a5ee1cc52",
"zenitram-malla-15753fae1999978e",
"zenitram-malla-45807117f798f314",
"zenitram-malla-4c7ce7ae0b9dd0cf",
"zenitram-malla-5b8f3116fefd595b",
"zenitram-malla-72b8d4dc93a566fb",
"zenitram-malla-73c0175a1009ffbf",
"zenitram-malla-825a0f3ce563c373",
"zenitram-malla-858255cb6145e104",
"zenitram-malla-92e437caca55290b",
"zenitram-malla-aeb1d1e3d40815a8",
"zenitram-malla-bf4b3bcd2652952c",
"zenitram-malla-c3ab7f5f4e769cc6",
"zenitram-malla-f3535c03a02fa0e3",
"zereight-gitlab-mcp-2367b459f8c3802b",
"zereight-gitlab-mcp-768c7d5a32ccc95b"
],
"rate": null,
"status": "not yet proved",
"wilson95": [
null,
null
]
},
"false_alarms_per_100_pushes": {
"cluster95": [
47.89915966386555,
62.916666666666664
],
"count": 132,
"icc": 0.02916674829529659,
"n": 238,
"rate": 55.46218487394958,
"wilson95": [
49.11079301666095,
61.64004554874716
]
},
"headline": "On unseen app repos: recall 41.7% (Wilson 19.3-68.0), false alarms 55.5 per 100 pushes, precision 3.5%; exploit-verified 0 of 26 (26 not yet proved).",
"precision": {
"count": 5,
"n": 141,
"rate": 0.03546099290780142,
"wilson95": [
0.015239951514497675,
0.08032375394028879
]
},
"recall": {
"cluster95": [
0.16666666666666666,
0.6666666666666666
],
"count": 5,
"n": 12,
"rate": 0.4166666666666667,
"wilson95": [
0.19325746190524656,
0.6804926643446272
]
}
}
benchmarks/measurements/2026-10-12-valid-benchmark-pilot/scored_a_interim_163.json → benchmarks/measurements/2026-10-12-valid-benchmark-pilot/scored_a.json
+65 -65
View File
@@ -1,11 +1,11 @@
{
"alerts_only": {
"count": 0,
"n": 149,
"n": 238,
"rate": 0.0,
"wilson95": [
0,
0.025134518351024853
1.734723475976807e-18,
0.015884777474181447
]
},
"breakdowns": {
@@ -13,11 +13,11 @@
"engine_covered": {
"alerts_only": {
"count": 0,
"n": 96,
"n": 112,
"rate": 0.0,
"wilson95": [
0,
0.03847694748481595
0.03316252537948371
]
},
"catch": {
@@ -32,26 +32,26 @@
"false_alarm": {
"cluster95": [
0.0,
0.033707865168539325
0.0297029702970297
],
"count": 1,
"icc": 2.814649921584905e-17,
"n": 96,
"rate": 0.010416666666666666,
"icc": -0.018481210769051442,
"n": 112,
"rate": 0.008928571428571428,
"wilson95": [
0.001841122818436635,
0.056667554927112326
0.0015778140883850773,
0.04884966619503642
]
}
},
"quick_only": {
"alerts_only": {
"count": 0,
"n": 53,
"n": 126,
"rate": 0.0,
"wilson95": [
0,
0.06758430445307662
0.029586819632536877
]
},
"catch": {
@@ -66,15 +66,15 @@
"false_alarm": {
"cluster95": [
0.0,
0.0
0.0625
],
"count": 0,
"icc": 0.0,
"n": 53,
"rate": 0.0,
"count": 4,
"icc": -0.02087087920452015,
"n": 126,
"rate": 0.031746031746031744,
"wilson95": [
0,
0.06758430445307662
0.012413078146214646,
0.07878727674774846
]
}
},
@@ -219,11 +219,11 @@
"unknown": {
"alerts_only": {
"count": 0,
"n": 149,
"n": 238,
"rate": 0.0,
"wilson95": [
0,
0.025134518351024853
1.734723475976807e-18,
0.015884777474181447
]
},
"catch": {
@@ -237,16 +237,16 @@
},
"false_alarm": {
"cluster95": [
0.0,
0.02027027027027027
0.004219409282700422,
0.041666666666666664
],
"count": 1,
"icc": -0.004445973772031825,
"n": 149,
"rate": 0.006711409395973154,
"count": 5,
"icc": 0.004133404058911234,
"n": 238,
"rate": 0.02100840336134454,
"wilson95": [
0.0011856820977550406,
0.037034278959967463
0.00900598570518914,
0.04822817086671577
]
}
},
@@ -289,11 +289,11 @@
"first_parent": {
"alerts_only": {
"count": 0,
"n": 149,
"n": 238,
"rate": 0.0,
"wilson95": [
0,
0.025134518351024853
1.734723475976807e-18,
0.015884777474181447
]
},
"catch": {
@@ -307,16 +307,16 @@
},
"false_alarm": {
"cluster95": [
0.0,
0.02027027027027027
0.004219409282700422,
0.041666666666666664
],
"count": 1,
"icc": -0.004445973772031825,
"n": 149,
"rate": 0.006711409395973154,
"count": 5,
"icc": 0.004133404058911234,
"n": 238,
"rate": 0.02100840336134454,
"wilson95": [
0.0011856820977550406,
0.037034278959967463
0.00900598570518914,
0.04822817086671577
]
}
},
@@ -359,11 +359,11 @@
"unknown": {
"alerts_only": {
"count": 0,
"n": 149,
"n": 238,
"rate": 0.0,
"wilson95": [
0,
0.025134518351024853
1.734723475976807e-18,
0.015884777474181447
]
},
"catch": {
@@ -377,16 +377,16 @@
},
"false_alarm": {
"cluster95": [
0.0,
0.02027027027027027
0.004219409282700422,
0.041666666666666664
],
"count": 1,
"icc": -0.004445973772031825,
"n": 149,
"rate": 0.006711409395973154,
"count": 5,
"icc": 0.004133404058911234,
"n": 238,
"rate": 0.02100840336134454,
"wilson95": [
0.0011856820977550406,
0.037034278959967463
0.00900598570518914,
0.04822817086671577
]
}
}
@@ -465,15 +465,15 @@
}
},
"coverage": {
"engine_covered": 100,
"engine_covered": 116,
"instrument_failure": 2,
"quick_only": 61,
"quick_only": 134,
"unanalysable": 0
},
"deadlines": [
{
"applied_deadline_scale": 1.0,
"count": 163,
"count": 252,
"deadline_scale": 1.0,
"deadline_seconds": 300,
"lane": "docker"
@@ -481,19 +481,19 @@
],
"false_alarm": {
"cluster95": [
0.0,
0.02027027027027027
0.004219409282700422,
0.041666666666666664
],
"count": 1,
"icc": -0.004445973772031825,
"n": 149,
"rate": 0.006711409395973154,
"count": 5,
"icc": 0.004133404058911234,
"n": 238,
"rate": 0.02100840336134454,
"wilson95": [
0.0011856820977550406,
0.037034278959967463
0.00900598570518914,
0.04822817086671577
]
},
"total": 163,
"total": 252,
"unseen": {
"gate": "not_run"
}
benchmarks/measurements/2026-10-12-valid-benchmark-pilot/scored_combined.json
+500
View File
@@ -0,0 +1,500 @@
{
"alerts_only": {
"count": 129,
"n": 238,
"rate": 0.542016806722689,
"wilson95": [
0.4785513166223543,
0.6041474415730925
]
},
"breakdowns": {
"coverage": {
"engine_covered": {
"alerts_only": {
"count": 61,
"n": 112,
"rate": 0.5446428571428571,
"wilson95": [
0.4524610601997964,
0.6338637143198924
]
},
"catch": {
"count": 2,
"n": 4,
"rate": 0.5,
"wilson95": [
0.15003570882017148,
0.8499642911798285
]
},
"false_alarm": {
"cluster95": [
0.48672566371681414,
0.6126126126126126
],
"count": 61,
"icc": -0.04646204088399717,
"n": 112,
"rate": 0.5446428571428571,
"wilson95": [
0.4524610601997964,
0.6338637143198924
]
}
},
"quick_only": {
"alerts_only": {
"count": 68,
"n": 126,
"rate": 0.5396825396825397,
"wilson95": [
0.4527676122693769,
0.6242493068074058
]
},
"catch": {
"count": 3,
"n": 8,
"rate": 0.375,
"wilson95": [
0.13684175946152105,
0.6942621454161981
]
},
"false_alarm": {
"cluster95": [
0.4605263157894737,
0.6557377049180327
],
"count": 71,
"icc": 0.044957643347841525,
"n": 126,
"rate": 0.5634920634920635,
"wilson95": [
0.47628498117338214,
0.6469420893494703
]
}
},
"unanalysable": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
}
},
"family": {
"access_control": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 0,
"n": 3,
"rate": 0.0,
"wilson95": [
0,
0.5615060804490177
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
},
"output_encoding": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 2,
"n": 4,
"rate": 0.5,
"wilson95": [
0.15003570882017148,
0.8499642911798285
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
},
"path": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 2,
"n": 3,
"rate": 0.6666666666666666,
"wilson95": [
0.2076549551264879,
0.9385096847238394
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
},
"unknown": {
"alerts_only": {
"count": 129,
"n": 238,
"rate": 0.542016806722689,
"wilson95": [
0.4785513166223543,
0.6041474415730925
]
},
"catch": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"false_alarm": {
"cluster95": [
0.4789915966386555,
0.6291666666666667
],
"count": 132,
"icc": 0.02916674829529659,
"n": 238,
"rate": 0.5546218487394958,
"wilson95": [
0.4911079301666095,
0.6164004554874716
]
}
},
"untrusted_destination": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 1,
"n": 2,
"rate": 0.5,
"wilson95": [
0.09452865480086614,
0.9054713451991339
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
}
},
"method": {
"first_parent": {
"alerts_only": {
"count": 129,
"n": 238,
"rate": 0.542016806722689,
"wilson95": [
0.4785513166223543,
0.6041474415730925
]
},
"catch": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"false_alarm": {
"cluster95": [
0.4789915966386555,
0.6291666666666667
],
"count": 132,
"icc": 0.02916674829529659,
"n": 238,
"rate": 0.5546218487394958,
"wilson95": [
0.4911079301666095,
0.6164004554874716
]
}
},
"fix_site": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 5,
"n": 12,
"rate": 0.4166666666666667,
"wilson95": [
0.19325746190524656,
0.6804926643446272
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
}
},
"post_cutoff": {
"unknown": {
"alerts_only": {
"count": 129,
"n": 238,
"rate": 0.542016806722689,
"wilson95": [
0.4785513166223543,
0.6041474415730925
]
},
"catch": {
"count": 5,
"n": 12,
"rate": 0.4166666666666667,
"wilson95": [
0.19325746190524656,
0.6804926643446272
]
},
"false_alarm": {
"cluster95": [
0.4789915966386555,
0.6291666666666667
],
"count": 132,
"icc": 0.02916674829529659,
"n": 238,
"rate": 0.5546218487394958,
"wilson95": [
0.4911079301666095,
0.6164004554874716
]
}
}
},
"stream": {
"advisory": {
"count": 0,
"n": 12,
"rate": 0.0,
"wilson95": [
0,
0.2425007574992425
]
},
"alerts": {
"count": 5,
"n": 12,
"rate": 0.4166666666666667,
"wilson95": [
0.19325746190524656,
0.6804926643446272
]
},
"quick": {
"count": 0,
"n": 12,
"rate": 0.0,
"wilson95": [
0,
0.2425007574992425
]
}
}
},
"catch": {
"count": 5,
"n": 12,
"rate": 0.4166666666666667,
"wilson95": [
0.19325746190524656,
0.6804926643446272
]
},
"confirmed": {
"alerts_only": {
"count": 0,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
},
"catch": {
"count": 5,
"n": 12,
"rate": 0.4166666666666667,
"wilson95": [
0.19325746190524656,
0.6804926643446272
]
},
"false_alarm": {
"cluster95": [
null,
null
],
"count": 0,
"icc": null,
"n": 0,
"rate": null,
"wilson95": [
null,
null
]
}
},
"coverage": {
"engine_covered": 116,
"instrument_failure": 2,
"quick_only": 134,
"unanalysable": 0
},
"deadlines": [
{
"applied_deadline_scale": 1.0,
"count": 252,
"deadline_scale": 1.0,
"deadline_seconds": 300,
"lane": "docker"
}
],
"false_alarm": {
"cluster95": [
0.4789915966386555,
0.6291666666666667
],
"count": 132,
"icc": 0.02916674829529659,
"n": 238,
"rate": 0.5546218487394958,
"wilson95": [
0.4911079301666095,
0.6164004554874716
]
},
"total": 252,
"unseen": {
"gate": "not_run"
}
}