marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ci): run ruff format, and record run 5 at 90% pair-correct

Browse Source 
CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.

Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.

Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:

                       hunter (K=5)   pipeline (K=1)
  pair-correct              76.7%          90.0%
  leak                      12.3%           6.7%
  detection                    --          96.7%
  cost                     ~$0.69        $0.0068

27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).

One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.

The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-08 19:46:58 +02:00
co-authored by Claude Opus 5
parent 14ca0a96ce
commit fe45081b43
24 changed files with 397 additions and 213 deletions
Download Patch File Download Diff File Expand all files Collapse all files
benchmarks/measurements/2026-09-08-pipeline-vibe-py.json
+26 -39
View File
@@ -1,47 +1,31 @@
{
"correction": "The three silent config pairs were reported as corpus defects on a similarity measure (23%, 45%, 53%). That threshold came from the OWASP minimal-fix analysis and does not apply to this slice: vibe-py pairs a vulnerable function with a DIFFERENT, SAFE function from the same repository, so low similarity is the design. djangoat-views-user-view is 36% similar and dsvpwa-handlers-do-bdr 34%, both pair_correct. Two of the three were a settings-table gap and now split.",
"cost_note": "Four full runs of this measurement cost roughly $0.04 in total.",
"cost_usd": 0.0067,
"detection_rate": 0.9333,
"division_of_labour": {
"model_corroborated": 2,
"model_entailed": 11,
"note": "The model arbitrates 13 of 28 findings, up from 9 of 24 in run 1. Both findings gained in run 4 came from the MODEL, not the LLM, and cost fell from $0.0126 to $0.0067 as work moved onto the free deterministic path. That is the payoff curve for fact-table coverage, measured: closing a gap moves findings out of the LLM's column into the model's, improving precision and cost at once.",
"suspicion": 15
},
"caveat": "30 pairs of ONE slice, and the slice iterated against all session. The +16% overfitting gap against vfc-js is why contributor-scan Req 8.1 and 8.2 exist. A contributor-facing tool has to hold up on a repository nobody tuned against, which is what that spec's Req 4 and 10 are for.",
"cost_usd": 0.0068,
"detection_rate": 0.9667,
"generated": "2026-09-08",
"hunter_baseline": {
"leak_rate": 0.123,
"note": "deepseek-v4-flash K=5, 2026-09-06-learning-families.json",
"pair_correct": 0.767
},
"known_limit": "permissive_values is a flat list of literals with no keyword association, so it holds both True and False. debug=True is dangerous while httponly=True is safe and the model cannot tell them apart. It did not bite here -- run 4's leak is identical to run 3's -- but it is a real limit of the fact schema.",
"leak_rate": 0.0667,
"model": "deepseek-v4-flash",
"outcomes": {
"both_flagged": 2,
"both_silent": 2,
"pair_correct": 26
"both_silent": 1,
"pair_correct": 27
},
"pair_correct_rate": 0.8667,
"pair_correct_rate": 0.9,
"pairs": 30,
"reading": "The wired pipeline beats the LLM hunter on both axes: 86.7% pair-correct against 76.7%, and 6.7% leak against 12.3% -- at one fifth of the model calls (K=1 against K=5) and ~100x cheaper ($0.0067 against ~$0.69). The earlier 'detection has regressed' reading was taken with the proposer disconnected, an arbiter judged against a detector, and before four wiring bugs were found.",
"remaining_misses": {
"both_flagged": {
"cause": "the leak: LLM proposals the model cannot contradict",
"family": "injection",
"pairs": 2
},
"dsvpwa-dsvpwa-main": "its setting is an argparse default (--ssl defaulting to 0), a CLI default rather than a security-setting call. Predicted before the settings change and unchanged by it.",
"vampi-users-update-password": "the dominance arbiter returns corroborated for it in isolation; still undiagnosed at pipeline level."
},
"reading": "The wired pipeline beats the LLM hunter on both axes: 90.0% pair-correct against 76.7%, and 6.7% leak against 12.3% -- at one fifth of the model calls (K=1 against K=5) and ~100x cheaper ($0.0068 against ~$0.69). 27 of 30 pairs correct, one miss remaining. The earlier 'detection has regressed' reading was taken with the proposer disconnected -- an arbiter judged against a detector -- and before five wiring bugs were found.",
"remaining_miss": "dsvpwa-dsvpwa-main: its setting is an argparse default (--ssl defaulting to 0), a CLI default rather than a security-setting call. Predicted out of reach before the settings work and still is; contorting the arbiter around one pair would be worse than the miss.",
"rows": [
{
"family": "injection",
"fixed_findings": 2,
"outcome": "both_flagged",
"pair": "damn-vulnerable-graphql-application-views-mutate",
"vuln_findings": 4,
"vuln_findings": 3,
"vuln_rungs": [
"suspicion"
]
@@ -51,7 +35,7 @@
"fixed_findings": 0,
"outcome": "pair_correct",
"pair": "damn-vulnerable-graphql-application-views-resolve-pastes",
"vuln_findings": 3,
"vuln_findings": 4,
"vuln_rungs": [
"suspicion"
]
@@ -119,7 +103,7 @@
"fixed_findings": 0,
"outcome": "pair_correct",
"pair": "dsvpwa-handlers-do-bdr",
"vuln_findings": 4,
"vuln_findings": 3,
"vuln_rungs": [
"suspicion"
]
@@ -227,10 +211,12 @@
{
"family": "access_control",
"fixed_findings": 0,
"outcome": "both_silent",
"outcome": "pair_correct",
"pair": "vampi-users-update-password",
"vuln_findings": 0,
"vuln_rungs": []
"vuln_findings": 1,
"vuln_rungs": [
"model_corroborated"
]
},
{
"family": "injection",
@@ -335,30 +321,31 @@
],
"run_history": [
{
"leak": 0.067,
"note": "access_control and config_secrets had no pipeline path to their arbiters",
"note": "absence and config families had no pipeline path to their arbiters",
"pair_correct": 0.733,
"run": 1
},
{
"leak": 0.067,
"note": "family routing fixed -- IDENTICAL result, which was the clue: the arbiter was reached and its verdict discarded",
"note": "routing fixed -- IDENTICAL result, which was the clue: the arbiter was reached and its verdict discarded",
"pair_correct": 0.733,
"run": 2
},
{
"leak": 0.067,
"note": "residual question phrased for the arbiter that raised it; 2 of 3 access_control pairs converted",
"note": "residual question phrased for the arbiter that raised it",
"pair_correct": 0.8,
"run": 3
},
{
"leak": 0.067,
"note": "aiohttp Application/RedisStorage added to the settings table; 2 more config pairs converted, leak unchanged",
"note": "aiohttp Application(debug=True) and RedisStorage(httponly=False) added to the settings table",
"pair_correct": 0.867,
"run": 4
},
{
"note": "the dominance question no longer contradicts its own evidence line",
"pair_correct": 0.9,
"run": 5
}
],
"seconds": 1029.0,
"seconds": 1213.5,
"slice": "vibe-py"
}
src/openultrasast/improve/evolve.py
+1 -3
View File
@@ -234,9 +234,7 @@ def run_round(
regressed,
small,
)
outcome = _outcome(
round_index, accepted=accepted, reason=reason, edits=edits, before=before, after=after, degradations=refusals
)
outcome = _outcome(round_index, accepted=accepted, reason=reason, edits=edits, before=before, after=after, degradations=refusals)
return RoundOutcome(
**{
**outcome.__dict__,
src/openultrasast/model/calibrate.py
+14 -9
View File
@@ -24,6 +24,7 @@ class FamilyGaps(TypedDict):
pairs: int
reasons: dict[str, int]
# Ascending, matching `ladder.Rung`. A pair is split when the vulnerable side outranks its fix.
_ORDER = {"none": 0, "": 0, "suspicion": 0, "model_corroborated": 1, "model_entailed": 2, "execution_confirmed": 3}
@@ -82,15 +83,18 @@ class CalibrationReport:
def to_markdown(self) -> str:
"""Per-slice rows first; there is no headline here that is not backed by them (Req 9.3, 11.1)."""
lines = ["## Model calibration", "", "Coverage is a pair the model *splits*: a strictly higher rung on",
"the vulnerable side than on its fix. Everything else is a named gap.", "",
"| slice | pairs | covered | gaps | coverage |", "|---|---|---|---|---|"]
lines = [
"## Model calibration",
"",
"Coverage is a pair the model *splits*: a strictly higher rung on",
"the vulnerable side than on its fix. Everything else is a named gap.",
"",
"| slice | pairs | covered | gaps | coverage |",
"|---|---|---|---|---|",
]
for slice_name, block in sorted(self.per_slice.items()):
pairs = block["pairs"] or 1
lines.append(
f"| {slice_name} | {block['pairs']} | {block['covered']} | {block['gaps']} | "
f"{block['covered'] / pairs:.0%} |"
)
lines.append(f"| {slice_name} | {block['pairs']} | {block['covered']} | {block['gaps']} | {block['covered'] / pairs:.0%} |")
by_family = self.gaps_by_family()
if by_family:
lines += ["", "### Gaps by family", ""]
@@ -116,8 +120,9 @@ def calibrate(outcomes: Iterable[PairOutcome]) -> CalibrationReport:
if split:
covered.append(outcome.pair)
else:
gaps.append(ModelGap(pair=outcome.pair, family=outcome.family, slice=outcome.slice,
reason=outcome.reason or _default_reason(outcome)))
gaps.append(
ModelGap(pair=outcome.pair, family=outcome.family, slice=outcome.slice, reason=outcome.reason or _default_reason(outcome))
)
return CalibrationReport(
covered=tuple(sorted(covered)),
gaps=tuple(sorted(gaps, key=lambda g: (g.family, g.pair))),
src/openultrasast/model/judge.py
+14 -6
View File
@@ -79,11 +79,14 @@ def judge(
said_vulnerable = _ask(client, model, spec, answer.witness, function)
if said_vulnerable is None:
return Verdict(rung=Rung.SUSPICION, family=spec.family, witness=answer.witness,
contradiction="the judge returned no usable answer")
return Verdict(rung=Rung.SUSPICION, family=spec.family, witness=answer.witness, contradiction="the judge returned no usable answer")
if not said_vulnerable:
return Verdict(rung=Rung.SUSPICION, family=spec.family, witness=answer.witness,
contradiction="the judge held the sanitizer on this path sufficient")
return Verdict(
rung=Rung.SUSPICION,
family=spec.family,
witness=answer.witness,
contradiction="the judge held the sanitizer on this path sufficient",
)
return answer # CORROBORATED: a real flow the judge agrees is not neutralised
@@ -96,8 +99,13 @@ def _ask(client: Any, model: str, spec: TaintSpec, witness: str, function: str)
line = tail.split(",")[0].strip(") ")
prompt = redact_secrets(
QUESTION.format(
language=spec.language, source=source.strip(), sink=sink.strip(), line=line or "?",
function=function or "?", sanitized="yes", family=spec.family,
language=spec.language,
source=source.strip(),
sink=sink.strip(),
line=line or "?",
function=function or "?",
sanitized="yes",
family=spec.family,
)
)
try:
src/openultrasast/model/pipeline.py
+3 -7
View File
@@ -64,9 +64,7 @@ class ModelFinding:
contradiction: str = ""
def _arbitrate(
cpg: CpgResult, spec: TaintSpec | DominanceSpec | ConfigSpec, *, function: str, parameter_sources: bool
) -> Verdict | None:
def _arbitrate(cpg: CpgResult, spec: TaintSpec | DominanceSpec | ConfigSpec, *, function: str, parameter_sources: bool) -> Verdict | None:
"""Route a family to the arbiter that can actually decide it.
Dispatching on the spec type rather than on a family name keeps this honest: a family without an arbiter
@@ -96,8 +94,7 @@ def scan_region(
if answer is not None and answer.rung is Rung.ENTAILED:
# The graph decided. No model call, and no candidate needs asking about: this region has its finding.
return [ModelFinding(site=_site(function, answer.witness), family=spec.family, rung=Rung.ENTAILED,
witness=answer.witness)]
return [ModelFinding(site=_site(function, answer.witness), family=spec.family, rung=Rung.ENTAILED, witness=answer.witness)]
if answer is not None:
# A real flow through a sanitizer. Whether it suffices is the residual, and the model HAS coverage
@@ -106,8 +103,7 @@ def scan_region(
return []
said = _ask(client, model, residual_question(spec, answer.witness, function))
if said:
return [ModelFinding(site=_site(function, answer.witness), family=spec.family,
rung=Rung.CORROBORATED, witness=answer.witness)]
return [ModelFinding(site=_site(function, answer.witness), family=spec.family, rung=Rung.CORROBORATED, witness=answer.witness)]
return []
# The model resolved nothing. It is SILENT, not negative — so the enumerator's candidates are put to the
src/openultrasast/model/report.py
+5 -4
View File
@@ -56,15 +56,16 @@ def render(rows: Sequence[SliceRow], *, development: str = "", deciding: str = "
total_pairs = sum(row.pairs for row in rows)
total_covered = sum(row.covered for row in rows)
lines += ["", f"Across every slice: {total_covered}/{total_pairs} = {total_covered / total_pairs:.0%} "
"— read the rows above it, not this line."]
lines += [
"",
f"Across every slice: {total_covered}/{total_pairs} = {total_covered / total_pairs:.0%} — read the rows above it, not this line.",
]
gap = overfitting_gap(rows, development=development, deciding=deciding)
if gap is not None:
lines += [
"",
f"**Overfitting gap: {gap:+.0%}** — `{development}` (development) minus `{deciding}`, "
f"which is the deciding real-world slice.",
f"**Overfitting gap: {gap:+.0%}** — `{development}` (development) minus `{deciding}`, which is the deciding real-world slice.",
]
elif development or deciding:
lines += ["", f"**Overfitting gap: not measurable** — `{development}` or `{deciding}` is absent from this run."]
src/openultrasast/model/specs.py
+1
View File
@@ -190,6 +190,7 @@ def taint_specs(
scoped = (facts if facts is not None else load_facts()).for_language(language)
families = taxonomy if taxonomy is not None else load_families()
sources = tuple(sorted({pattern for fact in scoped.sources for pattern in fact.patterns}))
# A shape qualifier means "this sink is safe in this form", never "this call cleans the value".
def _is_shape(fact: object) -> bool:
return bool(getattr(fact, "parameterized", False)) or getattr(fact, "literal_format_arg", None) is not None
src/openultrasast/model/taint.py
+13 -11
View File
@@ -67,17 +67,19 @@ def _flows(rows: object, *, function: str) -> list[dict[str, object]]:
continue
if function and not _in_scope(row, function):
continue
kept.append({
"sink": str(row.get("sink", "")),
"sinkArity": row.get("sinkArity"),
"sinkArg0Literal": row.get("sinkArg0Literal"),
"inLabeledScope": row.get("inLabeledScope"),
"sinkLine": str(row.get("sinkLine", "")),
"sinkMethod": str(row.get("sinkMethod", "")),
"source": str(row.get("source", "")),
"sanitized": bool(row.get("sanitized", False)),
"length": _as_int(row.get("length")),
})
kept.append(
{
"sink": str(row.get("sink", "")),
"sinkArity": row.get("sinkArity"),
"sinkArg0Literal": row.get("sinkArg0Literal"),
"inLabeledScope": row.get("inLabeledScope"),
"sinkLine": str(row.get("sinkLine", "")),
"sinkMethod": str(row.get("sinkMethod", "")),
"source": str(row.get("source", "")),
"sanitized": bool(row.get("sanitized", False)),
"length": _as_int(row.get("length")),
}
)
return kept
src/openultrasast/pairs.py
-1
View File
@@ -100,7 +100,6 @@ _DEFAULT_PROVENANCE = {"sast": "synthetic", "local": "human", "github": "human",
_MANIFEST_MECHANISM = "source_reaches_sink" # cheat-sheet fixtures referenced via expected_from
class CatalogError(ValueError):
"""Raised when a pair catalog violates the label schema."""
tests/test_cpg_backend.py
+15 -2
View File
@@ -44,8 +44,21 @@ def test_the_seam_imports_nothing_outside_the_standard_library() -> None:
imported.add(node.module.split(".")[0])
elif isinstance(node, ast.Import):
imported.update(alias.name.split(".")[0] for alias in node.names)
stdlib = {"__future__", "json", "os", "re", "shutil", "subprocess", "tempfile", "dataclasses", "pathlib", "typing",
"collections", "logging", "hashlib"}
stdlib = {
"__future__",
"json",
"os",
"re",
"shutil",
"subprocess",
"tempfile",
"dataclasses",
"pathlib",
"typing",
"collections",
"logging",
"hashlib",
}
assert imported <= stdlib, f"the seam imports outside the standard library: {sorted(imported - stdlib)}"
assert "import jpype" not in source and "py4j" not in source, "Joern is reached by subprocess, never in-process"
tests/test_index.py
-2
View File
@@ -100,5 +100,3 @@ def test_retrieval_package_is_bounded_by_character_budget() -> None:
assert package.role == "verifier"
assert package.hits
assert package.truncated is True
tests/test_model_calibrate.py
+14 -10
View File
@@ -48,12 +48,14 @@ def test_flagging_both_sides_equally_is_a_gap_not_a_success() -> None:
def test_gaps_are_grouped_by_family_with_their_reasons() -> None:
from openultrasast.model.calibrate import calibrate
report = calibrate([
_outcome("a", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("b", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("c", family="injection", vuln="none", reason="source_unmodelled"),
_outcome("d", family="injection"),
])
report = calibrate(
[
_outcome("a", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("b", family="path", vuln="none", reason="sink_not_in_table"),
_outcome("c", family="injection", vuln="none", reason="source_unmodelled"),
_outcome("d", family="injection"),
]
)
by_family = report.gaps_by_family()
assert set(by_family) == {"path", "injection"}
assert by_family["path"]["pairs"] == 2
@@ -65,10 +67,12 @@ def test_the_report_is_per_slice_and_names_the_deciding_one() -> None:
"""Req 9.3/11.1: the real-world slice decides; a development slice must not stand in for the corpus."""
from openultrasast.model.calibrate import calibrate
report = calibrate([
_outcome("a", slice_name="vibe-py"),
_outcome("b", slice_name="vfc-js", vuln="none", reason="sink_not_in_table"),
])
report = calibrate(
[
_outcome("a", slice_name="vibe-py"),
_outcome("b", slice_name="vfc-js", vuln="none", reason="sink_not_in_table"),
]
)
assert set(report.per_slice) == {"vibe-py", "vfc-js"}
assert report.per_slice["vibe-py"]["covered"] == 1
assert report.per_slice["vfc-js"]["gaps"] == 1
tests/test_model_config_value.py
+31 -17
View File
@@ -14,9 +14,9 @@ from pathlib import Path
def _spec(): # type: ignore[no-untyped-def]
from openultrasast.model.specs import ConfigSpec
return ConfigSpec(family="config_secrets", language="python",
settings=("CORS", "set_cookie", "app.run"),
permissive=("'*'", '"*"', "True", "0.0.0.0"))
return ConfigSpec(
family="config_secrets", language="python", settings=("CORS", "set_cookie", "app.run"), permissive=("'*'", '"*"', "True", "0.0.0.0")
)
def _cpg(rows): # type: ignore[no-untyped-def]
@@ -29,8 +29,7 @@ def test_a_permissive_literal_is_entailed() -> None:
from openultrasast.model.config_value import verdict
from openultrasast.model.ladder import Rung
rows = [{"setting": 'CORS(app, origins="*")', "line": "3", "method": "create_app",
"literalArgs": ['"*"'], "args": ["app", '"*"']}]
rows = [{"setting": 'CORS(app, origins="*")', "line": "3", "method": "create_app", "literalArgs": ['"*"'], "args": ["app", '"*"']}]
answer = verdict(_cpg(rows), _spec(), function="create_app")
assert answer is not None and answer.rung is Rung.ENTAILED
assert "*" in answer.witness and "CORS" in answer.witness
@@ -39,8 +38,15 @@ def test_a_permissive_literal_is_entailed() -> None:
def test_a_restrictive_literal_yields_no_finding() -> None:
from openultrasast.model.config_value import verdict
rows = [{"setting": 'CORS(app, origins="https://app.example")', "line": "3", "method": "create_app",
"literalArgs": ['"https://app.example"'], "args": ["app", '"https://app.example"']}]
rows = [
{
"setting": 'CORS(app, origins="https://app.example")',
"line": "3",
"method": "create_app",
"literalArgs": ['"https://app.example"'],
"args": ["app", '"https://app.example"'],
}
]
assert verdict(_cpg(rows), _spec(), function="create_app") is None
@@ -49,8 +55,15 @@ def test_a_non_literal_argument_is_corroborated_not_entailed() -> None:
from openultrasast.model.config_value import verdict
from openultrasast.model.ladder import Rung
rows = [{"setting": "CORS(app, origins=configured_origins)", "line": "3", "method": "create_app",
"literalArgs": [], "args": ["app", "configured_origins"]}]
rows = [
{
"setting": "CORS(app, origins=configured_origins)",
"line": "3",
"method": "create_app",
"literalArgs": [],
"args": ["app", "configured_origins"],
}
]
answer = verdict(_cpg(rows), _spec(), function="create_app")
assert answer is not None and answer.rung is Rung.CORROBORATED
@@ -72,8 +85,7 @@ def test_a_failed_query_is_not_read_as_a_safe_configuration() -> None:
def test_the_verdict_is_deterministic() -> None:
from openultrasast.model.config_value import verdict
rows = [{"setting": 'set_cookie("s", secure=False)', "line": "9", "method": "login",
"literalArgs": ["True", "0.0.0.0"], "args": ["x"]}]
rows = [{"setting": 'set_cookie("s", secure=False)', "line": "9", "method": "login", "literalArgs": ["True", "0.0.0.0"], "args": ["x"]}]
cpg = _cpg(rows)
assert verdict(cpg, _spec(), function="login") == verdict(cpg, _spec(), function="login")
@@ -90,8 +102,12 @@ def test_the_settings_table_covers_the_frameworks_in_the_corpus() -> None:
from openultrasast.model.specs import config_specs
settings = set(config_specs(language="python")["config_secrets"].settings)
for framework, call in (("aiohttp", "session_setup"), ("aiohttp", "EncryptedCookieStorage"),
("django", "SECURE_SSL_REDIRECT"), ("flask", "set_cookie")):
for framework, call in (
("aiohttp", "session_setup"),
("aiohttp", "EncryptedCookieStorage"),
("django", "SECURE_SSL_REDIRECT"),
("flask", "set_cookie"),
):
assert call in settings, f"{framework}'s {call} is not a modelled security setting"
@@ -102,8 +118,7 @@ def test_a_weak_algorithm_is_entailed_by_its_literal() -> None:
from openultrasast.model.specs import config_specs
spec = config_specs(language="python")["config_secrets"]
rows = [{"setting": 'hashlib.new("md5")', "line": "4", "method": "digest",
"literalArgs": ['"md5"'], "args": ['"md5"']}]
rows = [{"setting": 'hashlib.new("md5")', "line": "4", "method": "digest", "literalArgs": ['"md5"'], "args": ['"md5"']}]
answer = verdict(_cpg(rows), spec, function="digest")
assert answer is not None and answer.rung is Rung.ENTAILED
assert "md5" in answer.witness.lower()
@@ -114,6 +129,5 @@ def test_a_strong_algorithm_yields_no_finding() -> None:
from openultrasast.model.specs import config_specs
spec = config_specs(language="python")["config_secrets"]
rows = [{"setting": 'hashlib.new("sha256")', "line": "4", "method": "digest",
"literalArgs": ['"sha256"'], "args": ['"sha256"']}]
rows = [{"setting": 'hashlib.new("sha256")', "line": "4", "method": "digest", "literalArgs": ['"sha256"'], "args": ['"sha256"']}]
assert verdict(_cpg(rows), spec, function="digest") is None
tests/test_model_dominance.py
+1 -2
View File
@@ -31,8 +31,7 @@ def _cpg(rows): # type: ignore[no-untyped-def]
def _op(method: str, guards: list[str], line: int = 10) -> dict[str, object]:
return {"operation": "Note.query.filter_by(id=nid)", "opLine": str(line), "opMethod": method,
"dominatingGuards": guards}
return {"operation": "Note.query.filter_by(id=nid)", "opLine": str(line), "opMethod": method, "dominatingGuards": guards}
def test_an_unguarded_operation_among_guarded_siblings_is_entailed() -> None:
tests/test_model_judge.py
+25 -21
View File
@@ -19,8 +19,7 @@ from pathlib import Path
def _spec(): # type: ignore[no-untyped-def]
from openultrasast.model.specs import TaintSpec
return TaintSpec(family="injection", language="python", sources=("request.args",),
sinks=("os.system",), sanitizers=("escape",))
return TaintSpec(family="injection", language="python", sources=("request.args",), sinks=("os.system",), sanitizers=("escape",))
def _cpg(rows): # type: ignore[no-untyped-def]
@@ -29,8 +28,9 @@ def _cpg(rows): # type: ignore[no-untyped-def]
return CpgResult(cpg_path=Path("cpg.bin"), run=lambda q, p: rows)
ENTAILING_ROW = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run",
"source": "request.args['c']", "sanitized": False, "length": 3}]
ENTAILING_ROW = [
{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": False, "length": 3}
]
class _Client:
@@ -75,8 +75,7 @@ def test_a_claim_the_model_supports_advances_to_corroborated() -> None:
from openultrasast.model.judge import judge
from openultrasast.model.ladder import Rung
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run",
"source": "request.args['c']", "sanitized": True, "length": 3}]
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}]
client = _Client('{"vulnerable": true, "family": "injection"}')
answer = judge(_cpg(rows), _spec(), function="run", client=client, model="m")
assert answer.rung is Rung.CORROBORATED
@@ -88,8 +87,7 @@ def test_the_llm_declining_leaves_a_corroborated_flow_at_suspicion() -> None:
from openultrasast.model.judge import judge
from openultrasast.model.ladder import Rung
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run",
"source": "request.args['c']", "sanitized": True, "length": 3}]
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}]
client = _Client('{"vulnerable": false, "family": "injection"}')
answer = judge(_cpg(rows), _spec(), function="run", client=client, model="m")
assert answer.rung is Rung.SUSPICION
@@ -101,8 +99,9 @@ def test_without_a_client_the_verdict_still_stands_on_its_own() -> None:
from openultrasast.model.ladder import Rung
assert judge(_cpg(ENTAILING_ROW), _spec(), function="run", client=None, model="").rung is Rung.ENTAILED
sanitized = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run",
"source": "request.args['c']", "sanitized": True, "length": 3}]
sanitized = [
{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}
]
answer = judge(_cpg(sanitized), _spec(), function="run", client=None, model="")
assert answer.rung is Rung.SUSPICION and "endpoint" in answer.contradiction.lower()
@@ -111,8 +110,7 @@ def test_the_question_is_bounded_typed_and_offers_no_tools() -> None:
"""Req 8.1: the LLM answers one question about a site it was handed; it never searches for the site."""
from openultrasast.model.judge import judge
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run",
"source": "request.args['c']", "sanitized": True, "length": 3}]
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']", "sanitized": True, "length": 3}]
client = _Client()
judge(_cpg(rows), _spec(), function="run", client=client, model="m")
call = client.calls[0]
@@ -126,8 +124,16 @@ def test_the_prompt_is_redacted() -> None:
"""Req 11.4: every prompt passes through redaction."""
from openultrasast.model.judge import judge
rows = [{"sink": "connect(pw)", "sinkLine": "4", "sinkMethod": "run",
"source": "AKIA" + "IOSFODNN7" + "EXAMPLE", "sanitized": True, "length": 2}]
rows = [
{
"sink": "connect(pw)",
"sinkLine": "4",
"sinkMethod": "run",
"source": "AKIA" + "IOSFODNN7" + "EXAMPLE",
"sanitized": True,
"length": 2,
}
]
client = _Client()
judge(_cpg(rows), _spec(), function="run", client=client, model="m")
text = " ".join(str(m.get("content", "")) for m in client.calls[0]["messages"]) # type: ignore[index,union-attr]
@@ -142,13 +148,11 @@ def test_there_is_no_k_run_averaging_anywhere_in_the_judge() -> None:
tree = ast.parse(source)
# Check CODE, not prose: the docstring is allowed to name the machinery it replaced.
names = {
node.id for node in ast.walk(tree) if isinstance(node, ast.Name)
} | {
node.attr for node in ast.walk(tree) if isinstance(node, ast.Attribute)
} | {
node.name for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)
}
names = (
{node.id for node in ast.walk(tree) if isinstance(node, ast.Name)}
| {node.attr for node in ast.walk(tree) if isinstance(node, ast.Attribute)}
| {node.name for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)}
)
for banned in ("k_runs", "majority", "vote", "votes", "average", "mean", "acceptance", "floor", "budget"):
assert banned not in names, f"{banned} has no place in the judge"
tests/test_model_ladder.py
+13 -3
View File
@@ -15,9 +15,19 @@ def _finding(**kw: object): # type: ignore[no-untyped-def]
from openultrasast.findings import StaticFinding
base = dict(
finding_id="r1:app.py:1", path="app.py", title="t", severity="high", confidence="medium",
evidence_level="static_corroboration", rationale="why", line=1, function_name="run",
reachability_status="unknown", reachability_evidence=[], reachability_conditions=[], tags=[],
finding_id="r1:app.py:1",
path="app.py",
title="t",
severity="high",
confidence="medium",
evidence_level="static_corroboration",
rationale="why",
line=1,
function_name="run",
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
)
base.update(kw)
tests/test_model_pipeline.py
+52 -21
View File
@@ -24,8 +24,7 @@ from pathlib import Path
def _spec(): # type: ignore[no-untyped-def]
from openultrasast.model.specs import TaintSpec
return TaintSpec(family="injection", language="python", sources=("request.args",),
sinks=("os.system",), sanitizers=("escape",))
return TaintSpec(family="injection", language="python", sources=("request.args",), sinks=("os.system",), sanitizers=("escape",))
def _cpg(rows): # type: ignore[no-untyped-def]
@@ -46,10 +45,28 @@ class _Client:
return ChatResponse(content=self._answer)
ENTAILING = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']",
"sanitized": False, "length": 3, "inLabeledScope": True}]
SANITIZED = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "request.args['c']",
"sanitized": True, "length": 3, "inLabeledScope": True}]
ENTAILING = [
{
"sink": "os.system(cmd)",
"sinkLine": "4",
"sinkMethod": "run",
"source": "request.args['c']",
"sanitized": False,
"length": 3,
"inLabeledScope": True,
}
]
SANITIZED = [
{
"sink": "os.system(cmd)",
"sinkLine": "4",
"sinkMethod": "run",
"source": "request.args['c']",
"sanitized": True,
"length": 3,
"inLabeledScope": True,
}
]
def test_an_entailed_site_becomes_a_finding_with_no_model_call() -> None:
@@ -85,8 +102,14 @@ def test_a_candidate_the_model_cannot_see_is_reported_at_suspicion() -> None:
from openultrasast.model.pipeline import scan_region
client = _Client()
findings = scan_region(_cpg([]), _spec(), function="run", client=client, model="m",
candidates=({"id": "app.py:9:raw_query", "text": "db.raw(q)", "line": 9},))
findings = scan_region(
_cpg([]),
_spec(),
function="run",
client=client,
model="m",
candidates=({"id": "app.py:9:raw_query", "text": "db.raw(q)", "line": 9},),
)
assert client.calls == 1, "one bounded question per candidate"
assert len(findings) == 1 and findings[0].rung is Rung.SUSPICION
assert findings[0].site == "app.py:9:raw_query"
@@ -96,8 +119,14 @@ def test_a_candidate_the_judge_declines_is_not_reported() -> None:
from openultrasast.model.pipeline import scan_region
client = _Client('{"vulnerable": false, "why": "constant"}')
findings = scan_region(_cpg([]), _spec(), function="run", client=client, model="m",
candidates=({"id": "app.py:9:x", "text": "db.raw('literal')", "line": 9},))
findings = scan_region(
_cpg([]),
_spec(),
function="run",
client=client,
model="m",
candidates=({"id": "app.py:9:x", "text": "db.raw('literal')", "line": 9},),
)
assert findings == []
@@ -107,8 +136,9 @@ def test_without_a_client_only_what_the_model_establishes_is_reported() -> None:
from openultrasast.model.pipeline import scan_region
assert scan_region(_cpg(ENTAILING), _spec(), function="run", client=None, model="", candidates=())[0].rung is Rung.ENTAILED
assert scan_region(_cpg([]), _spec(), function="run", client=None, model="",
candidates=({"id": "a:1:x", "text": "t", "line": 1},)) == []
assert (
scan_region(_cpg([]), _spec(), function="run", client=None, model="", candidates=({"id": "a:1:x", "text": "t", "line": 1},)) == []
)
def test_the_candidate_budget_is_bounded() -> None:
@@ -132,8 +162,7 @@ def test_the_candidate_budget_is_bounded() -> None:
def _dominance_spec(): # type: ignore[no-untyped-def]
from openultrasast.model.specs import DominanceSpec
return DominanceSpec(family="access_control", language="python",
operations=("filter_by",), dischargers=("current_user",))
return DominanceSpec(family="access_control", language="python", operations=("filter_by",), dischargers=("current_user",))
def _config_spec(): # type: ignore[no-untyped-def]
@@ -146,9 +175,10 @@ def test_an_absence_family_is_arbitrated_by_dominance_not_taint() -> None:
from openultrasast.model.ladder import Rung
from openultrasast.model.pipeline import scan_region
rows = [{"operation": "Note.query.filter_by(id=n)", "opLine": "9", "opMethod": "leaky", "dominatingGuards": []},
{"operation": "Note.query.filter_by(id=n)", "opLine": "3", "opMethod": "safe",
"dominatingGuards": ["current_user.id"]}]
rows = [
{"operation": "Note.query.filter_by(id=n)", "opLine": "9", "opMethod": "leaky", "dominatingGuards": []},
{"operation": "Note.query.filter_by(id=n)", "opLine": "3", "opMethod": "safe", "dominatingGuards": ["current_user.id"]},
]
findings = scan_region(_cpg(rows), _dominance_spec(), function="leaky", client=None, model="", candidates=())
assert len(findings) == 1 and findings[0].rung is Rung.ENTAILED
assert findings[0].family == "access_control"
@@ -158,8 +188,7 @@ def test_a_configuration_family_is_arbitrated_by_constant_abstraction() -> None:
from openultrasast.model.ladder import Rung
from openultrasast.model.pipeline import scan_region
rows = [{"setting": 'CORS(app, origins="*")', "line": "3", "method": "create_app",
"literalArgs": ['"*"'], "args": ["app"]}]
rows = [{"setting": 'CORS(app, origins="*")', "line": "3", "method": "create_app", "literalArgs": ['"*"'], "args": ["app"]}]
findings = scan_region(_cpg(rows), _config_spec(), function="create_app", client=None, model="", candidates=())
assert len(findings) == 1 and findings[0].rung is Rung.ENTAILED
assert findings[0].family == "config_secrets"
@@ -191,8 +220,10 @@ def test_a_corroborated_absence_finding_survives_a_judge_that_agrees() -> None:
from openultrasast.model.ladder import Rung
from openultrasast.model.pipeline import scan_region
rows = [{"operation": "q.filter_by(id=n)", "opLine": "9", "opMethod": "leaky", "dominatingGuards": []},
{"operation": "q.filter_by(id=n)", "opLine": "3", "opMethod": "safe", "dominatingGuards": ["current_user"]}]
rows = [
{"operation": "q.filter_by(id=n)", "opLine": "9", "opMethod": "leaky", "dominatingGuards": []},
{"operation": "q.filter_by(id=n)", "opLine": "3", "opMethod": "safe", "dominatingGuards": ["current_user"]},
]
findings = scan_region(_cpg(rows), _dominance_spec(), function="leaky", client=_Client(), model="m", candidates=())
assert len(findings) == 1 and findings[0].rung is Rung.ENTAILED
tests/test_model_report.py
+15 -4
View File
@@ -62,10 +62,21 @@ def test_every_finding_shows_its_rung() -> None:
from openultrasast.model.report import render_findings
finding = StaticFinding(
finding_id="r:app.py:1", path="app.py", title="t", severity="high", confidence="medium",
evidence_level="static_corroboration", rationale="why", line=1, function_name="run",
reachability_status="unknown", reachability_evidence=[], reachability_conditions=[], tags=[],
ranking_priority=1.0, rung=Rung.ENTAILED,
finding_id="r:app.py:1",
path="app.py",
title="t",
severity="high",
confidence="medium",
evidence_level="static_corroboration",
rationale="why",
line=1,
function_name="run",
reachability_status="unknown",
reachability_evidence=[],
reachability_conditions=[],
tags=[],
ranking_priority=1.0,
rung=Rung.ENTAILED,
)
text = render_findings([finding])
assert "model_entailed" in text and "app.py" in text
tests/test_model_specs.py
+10 -3
View File
@@ -26,7 +26,7 @@ def test_the_guard_classifier_still_recognises_each_kind() -> None:
assert classify_guard_text('cur.execute("SELECT * FROM t WHERE id = %s", (uid,))') == "parameterized_call"
assert classify_guard_text("if not current_user.is_authenticated: abort(403)") == "auth_check"
assert classify_guard_text('if name not in ALLOWED: abort(400)') == "allowlist_test"
assert classify_guard_text("if name not in ALLOWED: abort(400)") == "allowlist_test"
assert classify_guard_text("if len(buf) > 64: return") == "bounds_test"
assert classify_guard_text("if value is None: return") == "null_test"
assert classify_guard_text("page = int(raw)") == "type_change"
@@ -50,8 +50,15 @@ def test_the_seeded_families_and_the_named_gaps_are_both_exact() -> None:
seeded |= {spec.family for spec in dominance_specs(language=language).values()}
deferred = {"unknown", "memory"} # the declined bucket and the deferred execution tier
assert seeded == {
"injection", "deserialization", "config_secrets", "access_control",
"path", "output_encoding", "untrusted_destination", "prototype", "memory",
"injection",
"deserialization",
"config_secrets",
"access_control",
"path",
"output_encoding",
"untrusted_destination",
"prototype",
"memory",
}
gaps = {family.id for family in taxonomy.families} - seeded - deferred
assert gaps == set(), f"a family lost its seed: {sorted(gaps)}"
tests/test_model_taint.py
+124 -38
View File
@@ -71,8 +71,12 @@ def test_the_verdict_is_restricted_to_the_labeled_function() -> None:
rows = [
{
"sink": "os.system(x)", "sinkLine": "40", "sinkMethod": "elsewhere",
"source": "request.args['x']", "sanitized": False, "length": 2,
"sink": "os.system(x)",
"sinkLine": "40",
"sinkMethod": "elsewhere",
"source": "request.args['x']",
"sanitized": False,
"length": 2,
}
]
assert verdict(_cpg(rows), _spec(), function="run") is None, "a flow in another function does not judge this one"
@@ -141,10 +145,18 @@ def _shape_spec(): # type: ignore[no-untyped-def]
def test_a_parameterised_sink_call_is_not_a_vulnerability_even_though_the_flow_reaches_it() -> None:
from openultrasast.model.taint import verdict
rows = [{
"sink": 'db.execute("select ... = %s", (name,))', "sinkLine": "5", "sinkMethod": "run",
"source": "request.args['name']", "sanitized": False, "length": 4, "sinkArity": 2, "sinkArg0Literal": True,
}]
rows = [
{
"sink": 'db.execute("select ... = %s", (name,))',
"sinkLine": "5",
"sinkMethod": "run",
"source": "request.args['name']",
"sanitized": False,
"length": 4,
"sinkArity": 2,
"sinkArg0Literal": True,
}
]
assert verdict(_cpg(rows), _shape_spec(), function="run") is None, "a bound query is the fix, not the bug"
@@ -152,10 +164,18 @@ def test_an_interpolated_sink_call_of_the_same_name_is_entailed() -> None:
from openultrasast.model.ladder import Rung
from openultrasast.model.taint import verdict
rows = [{
"sink": 'db.execute("select ... \'" + name + "\'")', "sinkLine": "5", "sinkMethod": "run",
"source": "request.args['name']", "sanitized": False, "length": 4, "sinkArity": 1, "sinkArg0Literal": False,
}]
rows = [
{
"sink": 'db.execute("select ... \'" + name + "\'")',
"sinkLine": "5",
"sinkMethod": "run",
"source": "request.args['name']",
"sanitized": False,
"length": 4,
"sinkArity": 1,
"sinkArg0Literal": False,
}
]
answer = verdict(_cpg(rows), _shape_spec(), function="run")
assert answer is not None and answer.rung is Rung.ENTAILED
@@ -165,10 +185,18 @@ def test_the_shape_test_only_applies_to_sinks_declared_safe_in_that_shape() -> N
from openultrasast.model.ladder import Rung
from openultrasast.model.taint import verdict
rows = [{
"sink": "os.system(full)", "sinkLine": "7", "sinkMethod": "run",
"source": "request.args['cmd']", "sanitized": False, "length": 5, "sinkArity": 1, "sinkArg0Literal": False,
}]
rows = [
{
"sink": "os.system(full)",
"sinkLine": "7",
"sinkMethod": "run",
"source": "request.args['cmd']",
"sanitized": False,
"length": 5,
"sinkArity": 1,
"sinkArg0Literal": False,
}
]
answer = verdict(_cpg(rows), _spec(), function="run") # os.system is not in safe_shape_sinks
assert answer is not None and answer.rung is Rung.ENTAILED
@@ -178,8 +206,7 @@ def test_a_row_without_shape_fields_is_judged_on_the_flow_alone() -> None:
from openultrasast.model.ladder import Rung
from openultrasast.model.taint import verdict
rows = [{"sink": "db.execute(q)", "sinkLine": "5", "sinkMethod": "run", "source": "request.args['n']",
"sanitized": False, "length": 3}]
rows = [{"sink": "db.execute(q)", "sinkLine": "5", "sinkMethod": "run", "source": "request.args['n']", "sanitized": False, "length": 3}]
answer = verdict(_cpg(rows), _shape_spec(), function="run")
assert answer is not None and answer.rung is Rung.ENTAILED
@@ -225,10 +252,17 @@ def test_a_sink_in_a_nested_closure_is_in_scope_when_the_query_says_so() -> None
from openultrasast.model.ladder import Rung
from openultrasast.model.taint import verdict
rows = [{
"sink": "fs.readFileSync(possibleFilename)", "sinkLine": "6", "sinkMethod": "<lambda>0",
"source": "req", "sanitized": False, "length": 4, "inLabeledScope": True,
}]
rows = [
{
"sink": "fs.readFileSync(possibleFilename)",
"sinkLine": "6",
"sinkMethod": "<lambda>0",
"source": "req",
"sanitized": False,
"length": 4,
"inLabeledScope": True,
}
]
answer = verdict(_cpg(rows), _spec(), function="requestListener")
assert answer is not None and answer.rung is Rung.ENTAILED
assert "<lambda>0" not in answer.witness, "the witness names the flow, not the synthetic closure name"
@@ -237,10 +271,17 @@ def test_a_sink_in_a_nested_closure_is_in_scope_when_the_query_says_so() -> None
def test_a_sink_in_an_unrelated_function_stays_out_of_scope() -> None:
from openultrasast.model.taint import verdict
rows = [{
"sink": "fs.readFileSync('/etc/' + name)", "sinkLine": "11", "sinkMethod": "unrelated",
"source": "name", "sanitized": False, "length": 2, "inLabeledScope": False,
}]
rows = [
{
"sink": "fs.readFileSync('/etc/' + name)",
"sinkLine": "11",
"sinkMethod": "unrelated",
"source": "name",
"sanitized": False,
"length": 2,
"inLabeledScope": False,
}
]
assert verdict(_cpg(rows), _spec(), function="requestListener") is None
@@ -248,8 +289,9 @@ def test_rows_without_the_scope_flag_fall_back_to_the_method_name() -> None:
"""A row from an engine that cannot report scope must not silently become in-scope."""
from openultrasast.model.taint import verdict
rows = [{"sink": "os.system(x)", "sinkLine": "9", "sinkMethod": "<lambda>0",
"source": "request.args['x']", "sanitized": False, "length": 2}]
rows = [
{"sink": "os.system(x)", "sinkLine": "9", "sinkMethod": "<lambda>0", "source": "request.args['x']", "sanitized": False, "length": 2}
]
assert verdict(_cpg(rows), _spec(), function="run") is None
@@ -266,12 +308,33 @@ def test_a_flow_from_a_real_source_outranks_a_shorter_one_from_a_bare_parameter(
from openultrasast.model.taint import verdict
rows = [
{"sink": "fs.readFileSync(p)", "sinkLine": "6", "sinkMethod": "<lambda>0", "source": "res",
"sanitized": False, "length": 3, "inLabeledScope": True},
{"sink": "fs.readFileSync(p)", "sinkLine": "6", "sinkMethod": "<lambda>0", "source": "this",
"sanitized": False, "length": 7, "inLabeledScope": True},
{"sink": "fs.readFileSync(p)", "sinkLine": "6", "sinkMethod": "<lambda>0",
"source": "request.args['f']", "sanitized": False, "length": 11, "inLabeledScope": True},
{
"sink": "fs.readFileSync(p)",
"sinkLine": "6",
"sinkMethod": "<lambda>0",
"source": "res",
"sanitized": False,
"length": 3,
"inLabeledScope": True,
},
{
"sink": "fs.readFileSync(p)",
"sinkLine": "6",
"sinkMethod": "<lambda>0",
"source": "this",
"sanitized": False,
"length": 7,
"inLabeledScope": True,
},
{
"sink": "fs.readFileSync(p)",
"sinkLine": "6",
"sinkMethod": "<lambda>0",
"source": "request.args['f']",
"sanitized": False,
"length": 11,
"inLabeledScope": True,
},
]
answer = verdict(_cpg(rows), _spec(), function="requestListener")
assert answer is not None and answer.rung is Rung.ENTAILED
@@ -291,10 +354,24 @@ def test_a_modelled_source_flow_decides_even_when_a_parameter_flow_is_unsanitize
from openultrasast.model.taint import verdict
rows = [
{"sink": "fs.readFileSync(safe)", "sinkLine": "41", "sinkMethod": "<lambda>0", "source": "res",
"sanitized": False, "length": 3, "inLabeledScope": True},
{"sink": "fs.readFileSync(safe)", "sinkLine": "41", "sinkMethod": "<lambda>0",
"source": "request.args['f']", "sanitized": True, "length": 22, "inLabeledScope": True},
{
"sink": "fs.readFileSync(safe)",
"sinkLine": "41",
"sinkMethod": "<lambda>0",
"source": "res",
"sanitized": False,
"length": 3,
"inLabeledScope": True,
},
{
"sink": "fs.readFileSync(safe)",
"sinkLine": "41",
"sinkMethod": "<lambda>0",
"source": "request.args['f']",
"sanitized": True,
"length": 22,
"inLabeledScope": True,
},
]
answer = verdict(_cpg(rows), _spec(), function="requestListener")
assert answer is not None and answer.rung is Rung.CORROBORATED, "a sanitized real flow is not an entailment"
@@ -305,7 +382,16 @@ def test_parameter_flows_still_decide_when_no_modelled_source_reaches_the_sink()
from openultrasast.model.ladder import Rung
from openultrasast.model.taint import verdict
rows = [{"sink": "os.system(cmd)", "sinkLine": "4", "sinkMethod": "run", "source": "username",
"sanitized": False, "length": 2, "inLabeledScope": True}]
rows = [
{
"sink": "os.system(cmd)",
"sinkLine": "4",
"sinkMethod": "run",
"source": "username",
"sanitized": False,
"length": 2,
"inLabeledScope": True,
}
]
answer = verdict(_cpg(rows), _spec(), function="run")
assert answer is not None and answer.rung is Rung.ENTAILED
tests/test_pair_hygiene.py
+1 -2
View File
@@ -18,8 +18,7 @@ IDENTICAL_TWINS = {
# `owasp` added 2026-09-08: 40 generated BenchmarkJava xss pairs, the only harvested category the model was
# shown to distinguish before vendoring. This guard exists so a corpus change is never silent — updating it
# is the acknowledgement.
VENDORED_PER_SLICE = {"local": 3, "github": 6, "sast": 11, "vibe-py": 35, "vfc-js": 17, "vfc": 176,
"agent-vfc": 29, "owasp": 40}
VENDORED_PER_SLICE = {"local": 3, "github": 6, "sast": 11, "vibe-py": 35, "vfc-js": 17, "vfc": 176, "agent-vfc": 29, "owasp": 40}
def test_identical_twins_are_unscorable_and_no_row_is_dropped() -> None:
tests/test_pair_registration_context.py
+1 -2
View File
@@ -403,8 +403,7 @@ def test_the_vendored_count_is_unchanged_by_the_re_harvest() -> None:
per_slice: dict[str, int] = {}
for case in cases:
per_slice[case.slice] = per_slice.get(case.slice, 0) + 1
assert per_slice == {"agent-vfc": 29, "github": 6, "local": 3, "owasp": 40, "sast": 11, "vfc": 176,
"vfc-js": 17, "vibe-py": 35}
assert per_slice == {"agent-vfc": 29, "github": 6, "local": 3, "owasp": 40, "sast": 11, "vfc": 176, "vfc-js": 17, "vibe-py": 35}
def test_prose_inside_a_decorator_never_grants_a_guard(tmp_path: Path) -> None:
tests/test_semantic_pipeline.py
-2
View File
@@ -263,5 +263,3 @@ def test_markdown_report_includes_four_overlay_labels(tmp_path: Path) -> None:
assert "demoted" in text
assert "unadjudicated" in text
assert "coverage" in text
tests/test_subtraction_complete.py
+18 -4
View File
@@ -53,8 +53,14 @@ def test_no_surviving_module_imports_a_removed_one() -> None:
def test_the_removed_modules_are_gone_from_the_tree() -> None:
assert not (SRC / "learning").exists(), "the learning package is removed; taxonomy/candidates/endpoint live under model/"
for relative in ("semantic/mechanisms.py", "semantic/variants.py", "semantic/variant_search.py",
"semantic/prove_budget.py", "semantic/loo.py", "semantic/seed.py"):
for relative in (
"semantic/mechanisms.py",
"semantic/variants.py",
"semantic/variant_search.py",
"semantic/prove_budget.py",
"semantic/loo.py",
"semantic/seed.py",
):
assert not (SRC / relative).exists(), f"{relative} still exists"
@@ -69,8 +75,16 @@ def test_the_mechanism_lever_is_gone_from_the_retained_improve_loop() -> None:
"""
evolve = (SRC / "improve" / "evolve.py").read_text()
validator = (SRC / "improve" / "validator.py").read_text()
for symbol in ("MechanismEdit", "MechanismStore", "apply_mechanism_edits", "evaluate_mechanism_profiles",
"propose_mechanism_edits", "validate_mechanism", "mechanism_store", "mechanism_candidates"):
for symbol in (
"MechanismEdit",
"MechanismStore",
"apply_mechanism_edits",
"evaluate_mechanism_profiles",
"propose_mechanism_edits",
"validate_mechanism",
"mechanism_store",
"mechanism_candidates",
):
assert symbol not in evolve, f"{symbol} survives in improve/evolve.py"
assert symbol not in validator, f"{symbol} survives in improve/validator.py"
assert "def run_improvement(" in evolve and "def propose_status_edits(" in evolve, "the rule-status loop must survive"