mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(reports): say what was looked for, and what cannot be decided
contributor-scan 2.16, implementing the approved Req 5.6. `c/memory` scored zero true positives across 89k lines of C and three distinct classes of false one, for a structural reason: it models string CALLS and libpng overflows through arithmetic. The task allowed retiring the family. Kept, scoped and stated instead -- `strcpy` misuse is real in plenty of C and the family earns its place for that. Retiring it would have removed the disclosure along with the coverage, leaving a C scan silent for a reason nobody could read. **Its own description was the quiet-failure risk.** It claimed "bounds, lifetime, arithmetic", which is exactly what it does not decide. It now names the APIs it models, and a new `limits` field in families.toml says what it cannot: an overflow expressed as an index, a pointer offset, or a memcpy whose size is computed -- which is how most real C overflows are written, and how every one of libpng's is. A limit recorded in a fact table nobody reads is no better than silence, so the report renders it. **What was analysed** lists every family offered to a region whether or not it found anything, because a family that ran and reported nothing is the case a reader is most likely to misread, and it states plainly that silence means no fact table covered the code rather than that the code is safe. The counts are derived from the fact tables the arbiter actually used, so they cannot drift from what the engine looked for; only the limit sentence is authored, because "cannot decide an overflow expressed as an index" is a statement about an abstraction and no table can produce it. A C scan that finds nothing now says: | c | `memory` | 121 | 9 | **`memory` (c)** — Models CALLS, not arithmetic... Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
f488e625c2
commit
af7e33ef3e
@@ -122,3 +122,15 @@ def test_the_shipped_file_must_declare_the_whole_closed_set_in_order(tmp_path: P
|
||||
(tmp_path / "families.toml").write_text(GOOD)
|
||||
with pytest.raises(FamiliesError, match="in that order"):
|
||||
load_families(tmp_path / "families.toml")
|
||||
|
||||
|
||||
def test_the_memory_family_states_what_it_cannot_decide() -> None:
|
||||
"""contributor-scan 2.16. Its description used to claim "bounds, lifetime, arithmetic", which is exactly
|
||||
what it does not do -- the description was itself the quiet-failure risk."""
|
||||
from openultrasast.model.taxonomy import load_families
|
||||
|
||||
memory = next(family for family in load_families().families if family.id == "memory")
|
||||
|
||||
assert "arithmetic" in memory.limits, "the gap is stated"
|
||||
assert "arithmetic" not in memory.description, "and no longer claimed"
|
||||
assert "strcpy" in memory.description, "what it does cover is named"
|
||||
|
||||
@@ -517,3 +517,38 @@ def test_a_reasoned_claim_keeps_its_section_whatever_its_rung(tmp_path: Path) ->
|
||||
write_markdown_report(reasoned, output)
|
||||
|
||||
assert output.read_text().count(f"## {base.title}") == 20
|
||||
|
||||
|
||||
def test_the_report_says_what_was_looked_for_and_what_cannot_be_decided(tmp_path: Path) -> None:
|
||||
"""contributor-scan 2.16, Req 5.6. `c/memory` scored zero true positives across 89k lines of C for a
|
||||
structural reason -- it models string CALLS and libpng overflows through arithmetic. A silent scan
|
||||
reading as a clean bill of health is worse than a noisy one, because the reader acts on it."""
|
||||
output = tmp_path / "report.md"
|
||||
coverage = [
|
||||
{
|
||||
"language": "c",
|
||||
"family": "memory",
|
||||
"regions": 121,
|
||||
"modelled": 9,
|
||||
"description": "Misuse of unbounded string and format APIs",
|
||||
"limits": "Models CALLS, not arithmetic. It cannot decide an overflow expressed as an index.",
|
||||
}
|
||||
]
|
||||
|
||||
write_markdown_report([], output, coverage=coverage)
|
||||
|
||||
text = output.read_text()
|
||||
assert "## What was analysed" in text
|
||||
assert "| c | `memory` | 121 | 9 |" in text, "what it looked for, counted from the fact table itself"
|
||||
assert "### Stated limits" in text
|
||||
assert "cannot decide an overflow expressed as an index" in text
|
||||
assert "silence here means no fact table covered the code, not that the code is safe" in text
|
||||
|
||||
|
||||
def test_a_scan_that_made_no_claims_states_no_limits(tmp_path: Path) -> None:
|
||||
"""A report with no model layer has nothing to disclaim; the section would be noise."""
|
||||
output = tmp_path / "report.md"
|
||||
|
||||
write_markdown_report([_finding()], output)
|
||||
|
||||
assert "## What was analysed" not in output.read_text()
|
||||
|
||||
Reference in New Issue
Block a user