marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench(ranking): the position benchmark sets its own query ceiling; v2 partials in the brief

Browse Source 
A pass over every region of pmpro (22,315 pairs) took 255 s without the join facts and past the scan's
300 s default with them, and a timed-out pass answers nothing -- reported once as tiers={} and the static
order. mwwpform with carried: 292 -> 30.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-11 12:01:55 +02:00
co-authored by Claude Fable 5.1
parent 35cb578ce2
commit c8e2ed21aa
2 changed files with 14 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/ranking/position.py
+6
View File
@@ -16,6 +16,7 @@ from __future__ import annotations
import argparse
import json
import os
import sys
from collections import Counter
from pathlib import Path
@@ -48,6 +49,11 @@ def evidence_ordered_regions(root: Path, regions: list[ScanRegion], language: st
from openultrasast.cpg.backend import resolve_cpg_backend
from openultrasast.model.scan import _collect, _evidence_pass, _hook_callbacks, order_by_evidence
# A benchmark pass runs over EVERY region, not the scan's 500: pmpro's 22,315 pairs took 255 s without
# the join facts and past the scan's 300 s default with them, and a pass that times out answers
# nothing -- which read as "tiers={}" once and cost a run. The scan's per-push ceiling is not this
# measurement's; an operator who set one keeps it.
os.environ.setdefault("OPENULTRASAST_CPG_QUERY_TIMEOUT", "1800")
backend = resolve_cpg_backend()
started = time.monotonic()
# The RECIPE's language, never the first region's: WP Statistics's first region is JavaScript, and a