mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
feat(gate): Phase 4 task 8 — lock the detection gate into CI
Make the >=90% recall / <10% FP detection gate a first-class, score-independent CI step with non-regression guards. - openultrasast.gate: canonical corpus map + DetectionMetrics/DetectionGateVerdict, evaluate_corpus/detection_gate (no score input — cannot be unblocked by score), merge_gate (ANDs the hard detection gate with the advisory-first score gate), within_tolerance, and a `python -m openultrasast.gate` entrypoint that exits non-zero on a breach. Wired as an explicit step in both the GitHub and Dagger pipelines. - 8.1: pipeline tests prove a high/passing score cannot unblock a recall-breaching change; the gate fails below 90% recall or at/above 10% FP. - 8.2: tests/golden/detection_baseline.json + a byte-identical baseline test for the default ruleset / empty ledger (44/47 = 93.62% recall, 0% FP). - 8.3: within_tolerance band logic + a capability-gated HarnessX tolerance test. - 8.4: zero-dependency guard — governance/scoring/benchmark planes never import HarnessX (sys.modules check + module-scope import scan). - test_detection_benchmarks now sources the corpus/aggregation from the gate module. 128 passed, 3 skipped; ruff + mypy clean; detection gate PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
6e87a58465
commit
649bd42fc3
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "harnessx-self-improving-rulesets",
|
||||
"created_at": "2026-06-22T10:07:18Z",
|
||||
"updated_at": "2026-06-22T15:35:00Z",
|
||||
"updated_at": "2026-06-22T15:10:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -229,8 +229,8 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
|
||||
## Phase 4 — Validation: Cross-Cutting Gates & Non-Regression
|
||||
|
||||
- [ ] 8. Lock the detection-gate non-regression checks into CI
|
||||
- [ ] 8.1 Enforce the per-merge recall/FP detection gate
|
||||
- [x] 8. Lock the detection-gate non-regression checks into CI
|
||||
- [x] 8.1 Enforce the per-merge recall/FP detection gate
|
||||
- Evaluate aggregate recall and aggregate false-positive rate on the benchmark corpus on every merge and fail the merge if recall is below 90% or the false-positive rate is at or above 10%.
|
||||
- Treat the recall/FP gate as a hard constraint that is never folded into the project-score reward, blocking any change that breaches the gate regardless of score improvement.
|
||||
- Observable completion: a CI run fails the merge when recall drops below 90% or false-positive rate reaches 10%, and a score improvement cannot unblock a gate-breaching change, proven by pipeline tests.
|
||||
@@ -238,21 +238,21 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: CI Pipeline_
|
||||
- _Depends: 4.1_
|
||||
|
||||
- [ ] 8.2 (P) Assert byte-identical benchmark baseline for the zero-dep phases
|
||||
- [x] 8.2 (P) Assert byte-identical benchmark baseline for the zero-dep phases
|
||||
- With the default ruleset and an empty loop ledger, assert the benchmark output is byte-identical to the pre-feature baseline for the governance and rules-as-data phases.
|
||||
- Observable completion: a golden-diff test confirms byte-identical benchmark output with the default ruleset and empty ledger.
|
||||
- _Requirements: 6.2_
|
||||
- _Boundary: Benchmark Harness, CI Pipeline_
|
||||
- _Depends: 3.4, 4.3_
|
||||
|
||||
- [ ] 8.3 (P) Confirm HarnessX-backed stages stay within detection tolerance
|
||||
- [x] 8.3 (P) Confirm HarnessX-backed stages stay within detection tolerance
|
||||
- When a HarnessX-backed stage is enabled, confirm recall and false-positive rate stay within the configured tolerance of the equivalent zero-dependency baseline.
|
||||
- Observable completion: a CI run with a HarnessX-backed stage enabled keeps recall/FP within the configured tolerance of the baseline, proven by a tolerance test.
|
||||
- _Requirements: 6.4_
|
||||
- _Boundary: CI Pipeline, HxScanOrchestrator_
|
||||
- _Depends: 6.1, 8.1_
|
||||
|
||||
- [ ] 8.4 (P) Add the zero-dependency guard test
|
||||
- [x] 8.4 (P) Add the zero-dependency guard test
|
||||
- Run a guard test confirming that with HarnessX absent the governance, scoring, and benchmark planes operate and CI gating runs in the default install.
|
||||
- Observable completion: the guard test passes with HarnessX uninstalled and fails if any governance/scoring/benchmark path imports HarnessX.
|
||||
- _Requirements: 7.5_
|
||||
|
||||
Reference in New Issue
Block a user