marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-17 12:09:27 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(learning-harness): the gate baseline and the degradation matrix

Browse Source 
The three gate outputs are captured at 71efc9c -- the commit before the
first line of this feature -- and a test reproduces them byte for byte in
both matrices. Byte-identity was something a reviewer checked by hand
against a worktree; it is now mechanical and runs in CI.

Six absences are each exercised once with the reason they record: HarnessX,
the chat endpoint (the run still writes its artifact), the family
configurations, the network for pointer pairs, the semantic extra (named,
not guessed), and the sandbox -- so a claim nothing could run is not
confused with a claim that failed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 13:20:23 +02:00
co-authored by Claude Opus 5
parent 3e71e9e88b
commit 75ae9ee47b
3 changed files with 168 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-09-06-gate-baseline.txt
+22
View File
@@ -0,0 +1,22 @@
# Gate baseline for the learning-harness spec (task 5.1, Req 11.5).
# Captured at 71efc9c, the commit before the first line of this feature, by running in order:
# python -m openultrasast.gate
# python -m openultrasast.map_gate
# python -m openultrasast.pair_gate
# Every commit of this spec must reproduce this file byte for byte. The three gates are the only place
# where a detector change is allowed to be invisible: if any of them moves, something that was never
# supposed to change did.
# --- output begins ---
detection gate: recall 93.62% (44/47), FP 0.00% (0/44)
c_cpp recall 93.33% (14/15) FP 0.00% (0/14)
java recall 100.00% (4/4) FP 0.00% (0/4)
javascript recall 92.31% (12/13) FP 0.00% (0/12)
python recall 93.33% (14/15) FP 0.00% (0/14)
detection gate: PASS
split-sink map gate: PASS
pair gate: pair_pass 100.00% (3/3), vuln_recall 100.00%, silent_fix 100.00%, Youden +100.00%, labeled_recall 100.00% (28/28), fix_leaks 0
local pair_pass 100.00% (3/3) vuln 100.00% silent 100.00% Youden +100.00% labeled 100.00%
PASS local-python-vulnerable vuln 12/12 fix_findings=0 leaks=0
PASS local-javascript-vulnerable vuln 10/10 fix_findings=0 leaks=0
PASS local-cpp-buffer vuln 6/6 fix_findings=0 leaks=0
pair gate: PASS (local slice)