mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
Marius CiepluchandClaude Opus 4.8
649bd42fc3
Make the >=90% recall / <10% FP detection gate a first-class, score-independent CI step with non-regression guards. - openultrasast.gate: canonical corpus map + DetectionMetrics/DetectionGateVerdict, evaluate_corpus/detection_gate (no score input — cannot be unblocked by score), merge_gate (ANDs the hard detection gate with the advisory-first score gate), within_tolerance, and a `python -m openultrasast.gate` entrypoint that exits non-zero on a breach. Wired as an explicit step in both the GitHub and Dagger pipelines. - 8.1: pipeline tests prove a high/passing score cannot unblock a recall-breaching change; the gate fails below 90% recall or at/above 10% FP. - 8.2: tests/golden/detection_baseline.json + a byte-identical baseline test for the default ruleset / empty ledger (44/47 = 93.62% recall, 0% FP). - 8.3: within_tolerance band logic + a capability-gated HarnessX tolerance test. - 8.4: zero-dependency guard — governance/scoring/benchmark planes never import HarnessX (sys.modules check + module-scope import scan). - test_detection_benchmarks now sources the corpus/aggregation from the gate module. 128 passed, 3 skipped; ruff + mypy clean; detection gate PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>
45 lines
869 B
JSON
45 lines
869 B
JSON
{
|
|
"overall": {
|
|
"actual": 44,
|
|
"expected": 47,
|
|
"false_positive_rate": 0.0,
|
|
"false_positives": 0,
|
|
"matched": 44,
|
|
"recall": 0.93617
|
|
},
|
|
"per_language": {
|
|
"c_cpp": {
|
|
"actual": 14,
|
|
"expected": 15,
|
|
"false_positive_rate": 0.0,
|
|
"false_positives": 0,
|
|
"matched": 14,
|
|
"recall": 0.933333
|
|
},
|
|
"java": {
|
|
"actual": 4,
|
|
"expected": 4,
|
|
"false_positive_rate": 0.0,
|
|
"false_positives": 0,
|
|
"matched": 4,
|
|
"recall": 1.0
|
|
},
|
|
"javascript": {
|
|
"actual": 12,
|
|
"expected": 13,
|
|
"false_positive_rate": 0.0,
|
|
"false_positives": 0,
|
|
"matched": 12,
|
|
"recall": 0.923077
|
|
},
|
|
"python": {
|
|
"actual": 14,
|
|
"expected": 15,
|
|
"false_positive_rate": 0.0,
|
|
"false_positives": 0,
|
|
"matched": 14,
|
|
"recall": 0.933333
|
|
}
|
|
}
|
|
}
|