marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ranking): phase 0 -- the baseline, the evidence instrument, and a 7x taint speedup found on the way

Browse Source 
flow-aware-ranking phase 0: measure before changing anything.

THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:

    pmpro         4,463 regions  83.8% at 0.30   CVE at 390   budget_at_recall 391
    mwwpform        983 regions  81.7% at 0.30   CVE at 512   budget_at_recall 513
    wpstatistics  1,673 regions  75.3% at 0.30   CVE at 503   budget_at_recall 504
    vampi            25 regions                  BOLA at 15, 10; SQLI reached transitively

That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.

THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.

    2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
    tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%

Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.

AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:

    2,026.3s -> 287.0s      8,105 -> 1,148 ms per request      same 5,779 rows

Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.

Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.

869 tests pass; query compile test 4 passed; gates byte-identical.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-10 18:22:18 +02:00
co-authored by Claude Fable 5.1
parent bfc3d806ba
commit 88ada8e48c
10 changed files with 1000 additions and 345 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-09-08-module-audit.json
+369 -340
View File
File diff suppressed because it is too large Load Diff
benchmarks/ranking/baselines/2026-09-10-access-rank.json
+96
View File
@@ -0,0 +1,96 @@
[
{
"repo": "mwwpform",
"regions": 983,
"distinct_ranks": 5,
"largest_tier_share": 0.817,
"largest_tier_rank": 0.3,
"budget_at_recall": 513,
"known": [
{
"id": "CVE-2023-6559",
"family": "path",
"site": "classes/services/class.mail.php:259:_delete_files",
"position": 512,
"rank": 0.3,
"in_scope": true,
"reached_transitively": false
}
]
},
{
"repo": "pmpro",
"regions": 4463,
"distinct_ranks": 5,
"largest_tier_share": 0.838,
"largest_tier_rank": 0.3,
"budget_at_recall": 391,
"known": [
{
"id": "CVE-2023-23488",
"family": "injection",
"site": "classes/class.memberorder.php:936:getMemberOrderByCode",
"position": 390,
"rank": 0.3,
"in_scope": true,
"reached_transitively": false
}
]
},
{
"repo": "vampi",
"regions": 25,
"distinct_ranks": 5,
"largest_tier_share": 0.36,
"largest_tier_rank": 1.0,
"budget_at_recall": 16,
"known": [
{
"id": "VAmPI-SQLI",
"family": "injection",
"site": "models/user_model.py:73:get_user",
"position": null,
"rank": null,
"in_scope": true,
"reached_transitively": true
},
{
"id": "VAmPI-BOLA",
"family": "access_control",
"site": "api_views/users.py:187:update_password",
"position": 15,
"rank": 0.8,
"in_scope": true,
"reached_transitively": false
},
{
"id": "VAmPI-BOLA-BOOKS",
"family": "access_control",
"site": "api_views/books.py:51:get_by_title",
"position": 10,
"rank": 0.8,
"in_scope": true,
"reached_transitively": false
}
]
},
{
"repo": "wpstatistics",
"regions": 1673,
"distinct_ranks": 6,
"largest_tier_share": 0.753,
"largest_tier_rank": 0.3,
"budget_at_recall": 504,
"known": [
{
"id": "CVE-2022-25148",
"family": "injection",
"site": "includes/class-wp-statistics-pages.php:225:record",
"position": 503,
"rank": 0.3,
"in_scope": true,
"reached_transitively": false
}
]
}
]
benchmarks/ranking/position.py
+121
View File
@@ -0,0 +1,121 @@
"""Phase 0 of flow-aware-ranking: where does each pinned CVE sit in the order the scan actually examines?
The ranker's success metric is not "is the CVE in the budget" -- on Paid Memberships Pro it already is, at
position 390 of 4,463 -- but **how small the budget can be while every CVE stays in it**. That number is
`budget_at_recall`, and until it is written down nothing later can claim to have improved it.
This is the baseline. It runs no engine and asks no model: it is the region pipeline exactly as the scan
uses it, sorted exactly as the scan sorts it, with each known vulnerability's position read off. Offline,
against pinned checkouts, so two runs of one repository agree.
ousast-venv/bin/python benchmarks/ranking/position.py # every pinned recipe with a checkout
ousast-venv/bin/python benchmarks/ranking/position.py --json # the same, as an artifact
"""
from __future__ import annotations
import argparse
import json
import sys
from collections import Counter
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
from openultrasast.mapping import analyze_entry_points # noqa: E402
from openultrasast.model.regions import ScanRegion, regions_for # noqa: E402
from openultrasast.model.shipped import declared_sources # noqa: E402
from openultrasast.preprocess import preprocess_repository # noqa: E402
from openultrasast.repos import checkout_path, load_repo_recipes # noqa: E402
def ordered_regions(root: Path) -> list[ScanRegion]:
"""The regions in the order `scan_repository` examines them. One definition, imported, never copied."""
_, targets = preprocess_repository(root)
regions = regions_for(analyze_entry_points(root, targets), targets, shipped=declared_sources(root))
# The scan's own total order. If this drifts from scan.py the baseline measures the wrong thing, which
# is why the sort key is the one thing here worth a test against the driver.
return sorted(regions, key=lambda r: (not r.shipped, -r.rank, r.path, r.function or ""))
def measure(name: str, root: Path, known: list[dict[str, object]]) -> dict[str, object]:
regions = ordered_regions(root)
tiers = Counter(r.rank for r in regions)
largest = max(tiers.values()) if tiers else 0
rows: list[dict[str, object]] = []
for item in known:
# A known vulnerability names a file and a function. Match on both, path by suffix because a recipe
# writes it relative to the repository and a region may carry it the same way or deeper.
hit = next(
(i for i, r in enumerate(regions) if r.path.endswith(str(item["file"])) and r.function == item["function"]),
None,
)
rows.append(
{
"id": item["id"],
"family": item["family"],
"site": f"{item['file']}:{item['line']}:{item['function']}",
# Zero-based. None means no region is named after this function: the site is a SINK that an
# entry-point region elsewhere reaches through `callDepth` -- VAmPI's SQL injection lives in
# `models/user_model.py:get_user` and is found from the handler that calls it. Its true
# position is that handler's, which only the flow can say, so it is reported and left out of
# `budget_at_recall` rather than counted as a miss it is not.
"position": hit,
"rank": regions[hit].rank if hit is not None else None,
"in_scope": item["in_scope"],
"reached_transitively": hit is None,
}
)
positioned = [r["position"] for r in rows if r["position"] is not None and r["in_scope"]]
return {
"repo": name,
"regions": len(regions),
"distinct_ranks": len(tiers),
"largest_tier_share": round(largest / len(regions), 3) if regions else 0.0,
"largest_tier_rank": max(tiers, key=lambda k: tiers[k]) if tiers else None,
# The smallest examined-region budget that still holds every in-scope known vulnerability.
"budget_at_recall": (max(positioned) + 1) if positioned else None,
"known": rows,
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
parser.add_argument("--json", action="store_true", help="emit one JSON object per repository")
parser.add_argument("--repo", help="only this recipe")
args = parser.parse_args()
results: list[dict[str, object]] = []
for recipe in load_repo_recipes():
if args.repo and recipe.name != args.repo:
continue
if "detection" not in recipe.measures:
continue
root = checkout_path(recipe)
if not root.is_dir():
print(f"{recipe.name}: not checked out, skipping", file=sys.stderr)
continue
known = [
{"id": k.id, "family": k.family, "file": k.file, "function": k.function, "line": k.line, "in_scope": k.in_scope}
for k in recipe.known
]
results.append(measure(recipe.name, root, known))
if args.json:
print(json.dumps(results, indent=2))
return 0
for r in results:
print(
f"{r['repo']}: {r['regions']} regions, {r['distinct_ranks']} distinct ranks, "
f"{100 * float(r['largest_tier_share']):.1f}% share rank {r['largest_tier_rank']}"
)
for k in r["known"]:
where = "reached via another region" if k["position"] is None else f"position {k['position']} (rank {k['rank']})"
scope = "" if k["in_scope"] else " [out of scope]"
print(f" {k['id']:16} {where:28} {k['site']}{scope}")
print(f" budget_at_recall = {r['budget_at_recall']}")
return 0
if __name__ == "__main__":
raise SystemExit(main())