mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Marius CiepluchandClaude Opus 4.8
649bd42fc3
Make the >=90% recall / <10% FP detection gate a first-class, score-independent CI step with non-regression guards. - openultrasast.gate: canonical corpus map + DetectionMetrics/DetectionGateVerdict, evaluate_corpus/detection_gate (no score input — cannot be unblocked by score), merge_gate (ANDs the hard detection gate with the advisory-first score gate), within_tolerance, and a `python -m openultrasast.gate` entrypoint that exits non-zero on a breach. Wired as an explicit step in both the GitHub and Dagger pipelines. - 8.1: pipeline tests prove a high/passing score cannot unblock a recall-breaching change; the gate fails below 90% recall or at/above 10% FP. - 8.2: tests/golden/detection_baseline.json + a byte-identical baseline test for the default ruleset / empty ledger (44/47 = 93.62% recall, 0% FP). - 8.3: within_tolerance band logic + a capability-gated HarnessX tolerance test. - 8.4: zero-dependency guard — governance/scoring/benchmark planes never import HarnessX (sys.modules check + module-scope import scan). - test_detection_benchmarks now sources the corpus/aggregation from the gate module. 128 passed, 3 skipped; ruff + mypy clean; detection gate PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>
74 lines
2.3 KiB
YAML
74 lines
2.3 KiB
YAML
name: CI
|
|
|
|
on:
|
|
pull_request:
|
|
push:
|
|
branches: [main]
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
checks:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
- uses: astral-sh/setup-uv@v7
|
|
with:
|
|
enable-cache: true
|
|
- uses: actions/setup-python@v6
|
|
with:
|
|
python-version-file: .python-version
|
|
- run: uv sync --group dev
|
|
- run: uv run ruff format --check .
|
|
- run: uv run ruff check .
|
|
- run: uv run mypy src/openultrasast
|
|
- run: uv run pytest
|
|
- name: Detection gate (>=90% recall, <10% FP — hard, score-independent)
|
|
run: uv run python -m openultrasast.gate
|
|
- run: uv run python -m compileall src tests
|
|
|
|
dagger:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 20
|
|
env:
|
|
# Must match dagger-io (0.21.7) and dagger/_engine/_version.py CLI_VERSION
|
|
# so the engine image tag is deterministic. The dagger-for-github action
|
|
# runs `docker image inspect registry.dagger.io/engine:v${DAGGER_VERSION}`
|
|
# and skips its own (flaky) pull when the image is already in the local
|
|
# docker cache, so the pinned tag below MUST equal the action's version.
|
|
DAGGER_VERSION: 0.21.7
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
- uses: astral-sh/setup-uv@v7
|
|
with:
|
|
enable-cache: true
|
|
- name: Pre-pull dagger engine image (retry transient registry timeouts)
|
|
timeout-minutes: 8
|
|
run: |
|
|
set -euo pipefail
|
|
image="registry.dagger.io/engine:v${DAGGER_VERSION}"
|
|
attempts=5
|
|
for i in $(seq 1 "${attempts}"); do
|
|
echo "::group::Pull attempt ${i}/${attempts}: ${image}"
|
|
if timeout 90 docker pull "${image}"; then
|
|
echo "Pulled ${image} on attempt ${i}"
|
|
docker image inspect "${image}" >/dev/null
|
|
echo "::endgroup::"
|
|
exit 0
|
|
fi
|
|
echo "Pull attempt ${i} failed."
|
|
echo "::endgroup::"
|
|
if [ "${i}" -lt "${attempts}" ]; then
|
|
echo "Retrying in 20s..."
|
|
sleep 20
|
|
fi
|
|
done
|
|
echo "All ${attempts} pull attempts failed for ${image}" >&2
|
|
exit 1
|
|
- uses: dagger/dagger-for-github@v7
|
|
with:
|
|
version: 0.21.7
|
|
verb: run
|
|
args: uv run python dagger/ci.py
|