mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
feat(ranking): phase 0 -- the baseline, the evidence instrument, and a 7x taint speedup found on the way
flow-aware-ranking phase 0: measure before changing anything.
THE BASELINE, committed as benchmarks/ranking/baselines/2026-09-10-access-rank.json
from benchmarks/ranking/position.py, which sorts regions exactly as the scan does:
pmpro 4,463 regions 83.8% at 0.30 CVE at 390 budget_at_recall 391
mwwpform 983 regions 81.7% at 0.30 CVE at 512 budget_at_recall 513
wpstatistics 1,673 regions 75.3% at 0.30 CVE at 503 budget_at_recall 504
vampi 25 regions BOLA at 15, 10; SQLI reached transitively
That corrects the brief's own cross-assessment: "concentration, not inclusion"
was measured on pmpro alone. On the other two plugins the CVE sits OUTSIDE a
500-region budget and is never examined; the slices found those CVEs only
because a slice has too few regions to exclude anything.
THE INSTRUMENT: `evidenceOnly` mode on the taint query -- what is in scope,
which sink calls, what shape, whether a sanitizer wraps THEIR argument, whether
a source is local or reachable -- returning before reachableByFlows is ever
called. model/evidence.py turns its rows into a vector per (region, family)
and a tier; scan.py records every pair's tier on the result and acts on none of
it. Every later phase is measured against a record made by scans that did not
know they were being measured.
2,500 pairs in 98.9s incl. JVM load, ~40 ms per pair
tier 0 -- no sink of that family in reach: 2,161 of 2,500 = 86%
Two earlier attempts ran past an hour each with no payload, and both were my
fault: repository-wide walks repeated per request -- the sink scan, the
reachable-method BFS, familyInRepo. Memoised across the batch.
AND THE SAME FIX REACHES THE DATAFLOW MODE. sinkCalls and reachableMethods do
not depend on the family, and were recomputed for all 2,500 requests. The
identical 250-request set with both joins on:
2,026.3s -> 287.0s 8,105 -> 1,148 ms per request same 5,779 rows
Most of what 5.13 measured as "the base taint query" was never dataflow. A 7x
speedup with nothing about the analysis changed; both corrections recorded in
the specs. Combined with 86% tier 0, the whole-repository taint query that
timed out at 40 minutes is ~339 real requests at ~1.1s: about six minutes.
Also: the module audit manifest is regenerated (91 load-bearing, 7 standalone,
0 orphaned) because evidence.py is now imported by the scan, and the
failed-query test is order-independent since the evidence pass records its own
failure before taint's.
869 tests pass; query compile test 4 passed; gates byte-identical.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Fable 5.1
Claude Fable 5.1
parent
bfc3d806ba
commit
88ada8e48c
@@ -49,6 +49,7 @@
|
||||
boundedSinks: String = "",
|
||||
parameterSources: String = "false",
|
||||
fieldSources: String = "true",
|
||||
evidenceOnly: String = "false",
|
||||
hookCallbacks: String = "",
|
||||
dispatchApply: String = "",
|
||||
requests: String = "",
|
||||
@@ -65,6 +66,12 @@
|
||||
|
||||
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
||||
|
||||
// Per-family answers that do not depend on the region, kept across the whole batch.
|
||||
val familyInRepoMemo = scala.collection.mutable.Map.empty[String, Boolean]
|
||||
val sinkCandidatesMemo = scala.collection.mutable.Map.empty[String, List[io.shiftleft.codepropertygraph.generated.nodes.Call]]
|
||||
val reachableMemo = scala.collection.mutable.Map.empty[(String, String, Int), Set[String]]
|
||||
val methodSourceMemo = scala.collection.mutable.Map.empty[String, Boolean]
|
||||
|
||||
def rowsFor(
|
||||
sourcesS: String,
|
||||
sinksS: String,
|
||||
@@ -74,6 +81,7 @@
|
||||
functionS: String,
|
||||
paramSrc: String,
|
||||
fieldSrc: String,
|
||||
evidenceS: String,
|
||||
fileS: String,
|
||||
depthS: String,
|
||||
boundedS: String
|
||||
@@ -132,7 +140,9 @@
|
||||
// Bounded by `callDepth`, because "every method transitively reachable from an entry point" is most of a
|
||||
// repository, and an unbounded question is how this becomes slow again.
|
||||
val depth = scala.util.Try(depthS.toInt).getOrElse(0)
|
||||
lazy val reachableMethods: Set[String] =
|
||||
lazy val reachableMethods: Set[String] = reachableMemo.getOrElseUpdate((function, fileS, depth), computeReachable)
|
||||
|
||||
def computeReachable: Set[String] =
|
||||
if (depth <= 0 || labeledMethods.isEmpty) Set.empty
|
||||
else {
|
||||
var frontier = labeledMethods.toSet
|
||||
@@ -502,7 +512,66 @@
|
||||
byName || sanitizerNames.exists(n => code.startsWith(n + "(") || code.contains("'" + n + "'") || code.contains("\"" + n + "\""))
|
||||
}
|
||||
|
||||
def sinkCalls = cpg.call.filter(c => sinkNames.exists(n => sinkMatches(c, n))).filter(c => inScope(c.method))
|
||||
// The candidate sink calls for a FAMILY do not depend on the region, and scanning every call in the graph
|
||||
// for each of 2,487 requests was a large share of what made a request cost seconds -- in evidence mode,
|
||||
// which asks for no dataflow at all, it alone took a query past an hour. Scanned once per sink list and
|
||||
// kept for the batch; the per-request work is the scope filter over that list.
|
||||
def sinkCalls =
|
||||
sinkCandidatesMemo
|
||||
.getOrElseUpdate(sinksS, cpg.call.filter(c => sinkNames.exists(n => sinkMatches(c, n))).l)
|
||||
.filter(c => inScope(c.method))
|
||||
|
||||
// ---- EVIDENCE MODE: the same question without the dataflow (flow-aware-ranking, phase 0) --------------
|
||||
//
|
||||
// Tiering is the CPG query WITHOUT dataflow; arbitration is the query WITH it. Same graph, same facts, two
|
||||
// costs. This branch answers "what is in scope" -- which sink calls, what shape they take, whether a
|
||||
// cleansing call wraps THEIR argument, whether a source is local or reachable -- and returns before
|
||||
// `reachableByFlows` is ever called. Milliseconds per call, where a flow costs 6.63 seconds.
|
||||
//
|
||||
// Every field is stated in fact-table terms, so the vector means the same thing in every language the
|
||||
// graph can hold. A `summary` row is always emitted, even when there are no sinks, because "no sinks" is
|
||||
// tier 0 and must be distinguishable from "the query failed".
|
||||
if (evidenceS == "true") {
|
||||
// Whether a method holds a modelled source, memoised by full name across the batch: the same helper
|
||||
// is reachable from hundreds of regions, and re-walking its AST for each of them is what made a
|
||||
// no-dataflow query take minutes. `cpg.method.filter(...)` over every method per request was the
|
||||
// other half of that, replaced by a direct lookup of the reachable names.
|
||||
def methodHasSource(m: io.shiftleft.codepropertygraph.generated.nodes.Method): Boolean =
|
||||
methodSourceMemo.getOrElseUpdate(m.fullName, m.ast.isCall.exists(c => sourcePatterns.exists(p => c.code.contains(p))))
|
||||
val sinkList = sinkCalls.l
|
||||
val sourceLocal = labeledMethods.exists(methodHasSource)
|
||||
val sourceNear = depth > 0 && cpg.method.fullNameExact(reachableMethods.toSeq: _*).exists(methodHasSource)
|
||||
// Anywhere in the repository at all. Exact at every callDepth, and the cheapest cut there is: a family
|
||||
// with no sink anywhere cannot produce a flow for any region. Memoised on the sink list, because it is
|
||||
// the same answer for every request of one family and a repository-wide walk per request turned a
|
||||
// milliseconds-per-pair query into minutes.
|
||||
val familyInRepo = familyInRepoMemo.getOrElseUpdate(sinksS, cpg.call.exists(c => sinkNames.exists(n => sinkMatches(c, n))))
|
||||
val summary = ujson.Obj(
|
||||
"kind" -> "summary",
|
||||
"sinks" -> sinkList.size,
|
||||
"sourceLocal" -> sourceLocal,
|
||||
"sourceNear" -> sourceNear,
|
||||
"familyInRepo" -> familyInRepo
|
||||
)
|
||||
val perSink = sinkList.map { sink =>
|
||||
val realArgs = sink.argument.argumentIndexGt(0).l
|
||||
ujson.Obj(
|
||||
"kind" -> "sink",
|
||||
"sink" -> sink.code.take(200),
|
||||
"sinkLine" -> sink.lineNumber.getOrElse(-1).toString,
|
||||
"sinkMethod" -> sink.method.name,
|
||||
"sinkFile" -> sink.method.filename.split("/").last,
|
||||
"sinkArity" -> realArgs.size,
|
||||
"sinkArg0Literal" -> realArgs.headOption.map(a => a.isLiteral).getOrElse(false),
|
||||
// A sanitizer of this family is an ANCESTOR of an argument in the AST -- it wraps what flows in --
|
||||
// as opposed to merely appearing somewhere in the same function. This is the field a text scan
|
||||
// gets wrong, and the one that would have misranked CVE-2023-23488: `esc_sql` is in that function,
|
||||
// on a different statement.
|
||||
"cleansedOnCall" -> sink.argument.ast.l.exists(node => sanitizesHere(node))
|
||||
)
|
||||
}
|
||||
return summary :: perSink
|
||||
}
|
||||
|
||||
val rows = sinkCalls.l.flatMap { sink =>
|
||||
// Data flows into the arguments; asking the call node itself finds nothing.
|
||||
@@ -576,6 +645,7 @@
|
||||
def field(name: String): String = req.obj.get(name).map(_.str).getOrElse("")
|
||||
val paramSrc = req.obj.get("parameterSources").map(_.str).getOrElse("false")
|
||||
val fieldSrc = req.obj.get("fieldSources").map(_.str).getOrElse("true")
|
||||
val evidence = req.obj.get("evidenceOnly").map(_.str).getOrElse("false")
|
||||
id -> ujson.Arr(
|
||||
rowsFor(
|
||||
field("sources"),
|
||||
@@ -589,6 +659,7 @@
|
||||
field("function"),
|
||||
paramSrc,
|
||||
fieldSrc,
|
||||
evidence,
|
||||
field("file"),
|
||||
field("callDepth"),
|
||||
field("boundedSinks")
|
||||
@@ -604,7 +675,7 @@
|
||||
println(ujson.write(ujson.Obj.from(answers.toSeq :+ ("__census__" -> census))))
|
||||
} else {
|
||||
println(
|
||||
ujson.write(ujson.Arr(rowsFor(sources, sinks, sanitizers, hookCallbacks, dispatchApply, function, parameterSources, fieldSources, file, callDepth, boundedSinks): _*))
|
||||
ujson.write(ujson.Arr(rowsFor(sources, sinks, sanitizers, hookCallbacks, dispatchApply, function, parameterSources, fieldSources, evidenceOnly, file, callDepth, boundedSinks): _*))
|
||||
)
|
||||
}
|
||||
println("---OUSAST-CPG-END---")
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
"""The evidence vector: what a (region, family) pair carries BEFORE any dataflow is asked (flow-aware-ranking).
|
||||
|
||||
Tiering is the CPG query without dataflow; arbitration is the query with it. Same graph, same facts, two costs.
|
||||
This module turns the evidence rows that `taint.sc` emits in `evidenceOnly` mode into one vector per pair,
|
||||
and assigns the tier. Every field is stated in fact-table terms -- source, sink, sanitizer -- so a heuristic
|
||||
that mentions a language's own syntax has, by construction, left this module.
|
||||
|
||||
Only tier 0 excludes. Its predicate has a proof -- a family with no sink in reach cannot produce a flow, in any
|
||||
language, under any dataflow model -- and it is the only place exactness is required. Everything above orders,
|
||||
where being wrong costs position, and `regions_truncated` reports what position cost.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping, Sequence
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
TIER_EXCLUDE = 0 # no sink of this family in reach: never ask
|
||||
TIER_NO_SOURCE = 1 # sinks, but nothing untrusted can arrive: ask last
|
||||
TIER_CLEANSED = 2 # a source exists, but every sink call is bound or cleansed on the call: ask late
|
||||
TIER_OPEN = 3 # a source exists and some sink call is computed and uncleansed: ask first
|
||||
TIER_OPEN_PUBLIC = 4 # tier 3 and declared public, or a source on the same statement: ask very first
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SinkEvidence:
|
||||
"""One sink call of the family, as the graph saw it, with no flow asked."""
|
||||
|
||||
code: str
|
||||
line: int
|
||||
method: str
|
||||
arity: int
|
||||
arg0_literal: bool
|
||||
cleansed_on_call: bool # a sanitizer of this family wraps an ARGUMENT -- not merely present in the function
|
||||
|
||||
@property
|
||||
def shape(self) -> str:
|
||||
"""``literal`` cannot carry taint; ``computed`` is the residue that can."""
|
||||
return "literal" if self.arg0_literal else "computed"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Evidence:
|
||||
"""The vector for one (region, family) pair."""
|
||||
|
||||
sinks: tuple[SinkEvidence, ...] = ()
|
||||
family_in_repo: bool = True # False is the repository-wide tier 0, exact at every callDepth
|
||||
source_local: bool = False # a modelled source in the region's own body
|
||||
source_near: bool = False # a modelled source within the arbiter's callDepth
|
||||
entry: bool = False # parameters untrusted by contract
|
||||
access_declared_public: bool = False # declared, not inferred -- "no decorator found" is not "public"
|
||||
carried: bool | None = None # stage one of the two-stage join; None where it was not computed
|
||||
bound_names: tuple[str, ...] = field(default=()) # sink names the shape test treats as bound at arity >= 2
|
||||
|
||||
@property
|
||||
def any_source(self) -> bool:
|
||||
return self.source_local or self.source_near or self.entry or bool(self.carried)
|
||||
|
||||
def is_bound(self, sink: SinkEvidence) -> bool:
|
||||
"""The safe-shape rule the arbiter already applies: a named sink with two or more arguments binds."""
|
||||
return sink.arity >= 2 and any(name in sink.code for name in self.bound_names)
|
||||
|
||||
@property
|
||||
def open_sinks(self) -> tuple[SinkEvidence, ...]:
|
||||
"""Sink calls that could carry an uncleansed, unbound, computed value."""
|
||||
return tuple(s for s in self.sinks if s.shape == "computed" and not s.cleansed_on_call and not self.is_bound(s))
|
||||
|
||||
@property
|
||||
def tier(self) -> int:
|
||||
if not self.family_in_repo or not self.sinks:
|
||||
return TIER_EXCLUDE
|
||||
if not self.any_source:
|
||||
return TIER_NO_SOURCE
|
||||
if not self.open_sinks:
|
||||
return TIER_CLEANSED
|
||||
if self.access_declared_public or self.source_local or self.carried:
|
||||
return TIER_OPEN_PUBLIC
|
||||
return TIER_OPEN
|
||||
|
||||
|
||||
def evidence_from_rows(
|
||||
rows: Sequence[Mapping[str, object]] | None,
|
||||
*,
|
||||
entry: bool = False,
|
||||
access_declared_public: bool = False,
|
||||
carried: bool | None = None,
|
||||
bound_names: Sequence[str] = (),
|
||||
) -> Evidence | None:
|
||||
"""Read the query's evidence rows for one request. ``None`` when the query did not answer.
|
||||
|
||||
A request that answered with no sinks still carries a ``summary`` row -- that is tier 0, and it must be
|
||||
distinguishable from a request that failed, which carries nothing.
|
||||
"""
|
||||
if rows is None:
|
||||
return None
|
||||
summary = next((r for r in rows if isinstance(r, Mapping) and r.get("kind") == "summary"), None)
|
||||
if summary is None:
|
||||
return None
|
||||
sinks = tuple(
|
||||
SinkEvidence(
|
||||
code=str(r.get("sink", "")),
|
||||
line=_as_int(r.get("sinkLine")),
|
||||
method=str(r.get("sinkMethod", "")),
|
||||
arity=_as_int(r.get("sinkArity")),
|
||||
arg0_literal=_as_bool(r.get("sinkArg0Literal")),
|
||||
cleansed_on_call=_as_bool(r.get("cleansedOnCall")),
|
||||
)
|
||||
for r in rows
|
||||
if isinstance(r, Mapping) and r.get("kind") == "sink"
|
||||
)
|
||||
return Evidence(
|
||||
sinks=sinks,
|
||||
family_in_repo=_as_bool(summary.get("familyInRepo"), default=True),
|
||||
source_local=_as_bool(summary.get("sourceLocal")),
|
||||
source_near=_as_bool(summary.get("sourceNear")),
|
||||
entry=entry,
|
||||
access_declared_public=access_declared_public,
|
||||
carried=carried,
|
||||
bound_names=tuple(bound_names),
|
||||
)
|
||||
|
||||
|
||||
def _as_bool(value: object, *, default: bool = False) -> bool:
|
||||
"""A JSON boolean, or its string spelling. ``bool("False")`` is True, which would make every unsanitized
|
||||
sink read as cleansed the moment a serialiser chose strings -- the kind of silent inversion this project
|
||||
has paid for before."""
|
||||
if value is None:
|
||||
return default
|
||||
if isinstance(value, bool):
|
||||
return value
|
||||
return str(value).strip().lower() in {"true", "1", "yes"}
|
||||
|
||||
|
||||
def _as_int(value: object) -> int:
|
||||
try:
|
||||
return int(str(value))
|
||||
except (TypeError, ValueError):
|
||||
return -1
|
||||
|
||||
|
||||
__all__ = [
|
||||
"TIER_CLEANSED",
|
||||
"TIER_EXCLUDE",
|
||||
"TIER_NO_SOURCE",
|
||||
"TIER_OPEN",
|
||||
"TIER_OPEN_PUBLIC",
|
||||
"Evidence",
|
||||
"SinkEvidence",
|
||||
"evidence_from_rows",
|
||||
]
|
||||
@@ -26,6 +26,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from ..cpg.backend import CpgResult
|
||||
from .evidence import evidence_from_rows
|
||||
from .candidates import enumerate_candidates
|
||||
from .config_value import request_params as config_params
|
||||
from .dominance import request_params as dominance_params
|
||||
@@ -64,6 +65,11 @@ class ScanBudget:
|
||||
|
||||
max_model_calls: int = 200
|
||||
max_regions: int = 500
|
||||
# Compute the evidence tier of every (region, family) pair and RECORD it. Phase 0 of flow-aware-ranking:
|
||||
# the instrument runs on every real scan so later phases are measurable, and it changes nothing about
|
||||
# what is asked or in what order. One extra query invocation without dataflow -- a JVM load, then
|
||||
# milliseconds per pair.
|
||||
tiering: bool = True
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -87,6 +93,10 @@ class ModelScanResult:
|
||||
# would remove) or CPGQL evaluation (which it would not). That is the whole of the server-mode decision.
|
||||
query_seconds_by_kind: Mapping[str, float] = field(default_factory=dict)
|
||||
degradations: tuple[Mapping[str, object], ...] = ()
|
||||
# The evidence tier of each (path, function, family) pair the scan collected, and the count per tier.
|
||||
# Recorded, not acted on: this is what "the ranker got better" will be measured against.
|
||||
tiers: tuple[tuple[str, str, str, int], ...] = ()
|
||||
tier_counts: Mapping[int, int] = field(default_factory=dict)
|
||||
|
||||
@property
|
||||
def arbitrate_seconds(self) -> float:
|
||||
@@ -176,11 +186,44 @@ def scan_repository(
|
||||
continue
|
||||
work.append((f"{len(work)}", region, spec))
|
||||
|
||||
# Phase 1b: the evidence tier of every taint pair, before any dataflow is asked. Same query, `evidenceOnly`,
|
||||
# no `reachableByFlows`. Recorded on the result and NOT used to reorder or skip anything -- that is a
|
||||
# later phase, and it must be measured against this record before it is allowed to change behaviour.
|
||||
tiers: list[tuple[str, str, str, int]] = []
|
||||
per_kind: dict[str, float] = {}
|
||||
if limits.tiering and callable(getattr(cpg, "run_batch", None)):
|
||||
grouped_taint = _grouped(work, hook_callbacks=hooks).get("taint", {})
|
||||
if grouped_taint:
|
||||
evidence_requests = {rid: {**dict(params), "evidenceOnly": "true"} for rid, params in grouped_taint.items()}
|
||||
tier_started = time.monotonic()
|
||||
answered_evidence = cpg.run_batch("taint", evidence_requests)
|
||||
per_kind["evidence"] = round(time.monotonic() - tier_started, 2)
|
||||
if answered_evidence is None:
|
||||
degradations.append({"stage": "model", "reason": "query_failed", "kind": "evidence", "requests": len(evidence_requests)})
|
||||
else:
|
||||
answered_evidence.pop("__census__", None)
|
||||
by_rid = {rid: (region, spec) for rid, region, spec in work}
|
||||
for rid, rows in answered_evidence.items():
|
||||
pair = by_rid.get(rid)
|
||||
if pair is None:
|
||||
continue
|
||||
region, spec = pair
|
||||
evidence = evidence_from_rows(
|
||||
rows,
|
||||
entry=_is_entry_point(region),
|
||||
# Rank 1.0 is the public tier. Provenance -- declared by a route contract versus
|
||||
# inferred from an absent decorator -- is not yet carried on ScanRegion, so this is
|
||||
# the proxy phase 0 has; the brief names it as the field to thread through next.
|
||||
access_declared_public=region.rank >= 1.0,
|
||||
bound_names=getattr(spec, "safe_shape_sinks", ()),
|
||||
)
|
||||
if evidence is not None:
|
||||
tiers.append((region.path, region.function or "", getattr(spec, "family", ""), evidence.tier))
|
||||
|
||||
# Phase 2: ONE invocation per query kind. JVM startup dominates a repository scan -- a call per region
|
||||
# per family put a ten-line file at four minutes and a thousand regions at roughly fifty hours -- while
|
||||
# the queries themselves are milliseconds once the CPG is loaded.
|
||||
rows_by_id: dict[str, list[object]] = {}
|
||||
per_kind: dict[str, float] = {}
|
||||
census_reported = False
|
||||
query_started = time.monotonic()
|
||||
for kind, requests in _grouped(work, hook_callbacks=hooks).items():
|
||||
@@ -293,6 +336,8 @@ def scan_repository(
|
||||
query_seconds=query_seconds,
|
||||
query_seconds_by_kind=per_kind,
|
||||
degradations=tuple(degradations),
|
||||
tiers=tuple(tiers),
|
||||
tier_counts={tier: sum(1 for t in tiers if t[3] == tier) for tier in sorted({t[3] for t in tiers})},
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user