feat(reports): a report proportionate to what the engine can say

contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting
that `memcpy` and `strcpy` are PRESENT in a library that uses them
correctly throughout. That is the noise developers have learned to skip in
every other tool, and shipping it costs the findings beside it their
credibility.

**Filtering on rung was the obvious rule and it was wrong.** The
obligations checker emits at `suspicion` too, so hiding that rung would
have hidden an entire detector -- six tests failed and they were right to.
The discriminator is evidence_level: on libpng 163 of 198 rows are
`static_corroboration`, on vampi 3 of 60 are, and the model layer and the
obligations checker carry `suspicion` there instead. That is exactly the
distinction the ladder exists to make -- an undecided CLAIM about a site
is not an observation that an API exists.

So a pattern-only rule is grouped once it has more than three sites, and
keeps its full sections under that. One `python-flask-debug` is a finding a
contributor can act on and must not be summarised away; fifty-four
`c-unsafe-copy` are a property of the codebase.

  libpng   198 findings, 198 sections -> 39, 161 grouped into three lines
  vampi    60 findings, 61 sections -> 61, nothing grouped

Counted, never dropped: findings.json stays complete because it is the
record a baseline is compared against, and the grouped section says the
model reached none of these and that its silence means no fact table
covered the site rather than that the site is safe.

What it does not fix, recorded rather than glossed: vampi still emits 61
sections for a 520-line application, 51 of them reasoned claims this rule
does not touch. It fixes pattern noise, not volume.

Also records what group 2 established about the three corpus asks, so none
is started on a wrong assumption. Joern ships php2cpg but our fact tables
have zero PHP families, so WordPress would parse and produce nothing --
fact tables first, corpus second. An older vulnerable libpng will not help
for a reason no version can fix. Vibe-code repositories are the best fit
and the cheapest route to a second true-positive checkout, because Python
and JavaScript are the two languages whose fact tables are complete.

And Group 8, for per-repository state a developer owns: a committed
dismissal file the tool reads and never writes, dismissals as a split
evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION --
the judge and residual prompts -- never to the fact tables. That boundary
is the whole design: a prompt is how we ask, a fact table is what we
claim, and the closed-loop leak measured on 2026-09-05 is why the
distinction is not negotiable.

Gates byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/contributor-scan/tasks.md

@@ -197,7 +197,7 @@
   - _Requirements: 5.1, 8.1_
   - _Depends: 2.4_
 
-- [ ] 2.11 The suspicion band has to be worth reading
+- [x] 2.11 The suspicion band has to be worth reading
   - libpng's scan produced 198 rows: 26 entailed (all one false positive, since fixed), 113 `c-unsafe-*`
     pattern matches that assert an API is PRESENT rather than misused, 50 overlay-coverage annotations that
     are not findings at all, and 9 model suspicions. Zero validated defects.
@@ -347,6 +347,20 @@
 
 ## Group 5 — Grow the corpus
 
+**What group 2 established about each of these**, so none is started on a wrong assumption:
+
+* **PHP/WordPress**: Joern ships `php2cpg`, so the engine can parse it. The blocker is ours --
+  `taint_specs(language="php")` returns **zero families**, as do ruby and go. WordPress would parse and
+  produce nothing, which is the quiet failure that reads as a clean bill of health. The work is fact tables
+  first, corpus second.
+* **A known-vulnerable old libpng**: measured and it will not work, for a reason the version cannot fix.
+  Every file using `strcpy`/`strcat`/`sprintf`/`gets` is under `contrib/`; the library uses none of them and
+  overflows through arithmetic instead. See 2.15 and 2.16.
+* **Vibe-code repos**: the best fit available. Python and JavaScript both have full fact tables, the pair
+  corpus already carries 70 vibe-py and 17 vfc-js pairs, and vampi -- the one repository where the engine
+  finds 3 of 3 -- is exactly this shape. A whole-repository vibe-code checkout is the cheapest next
+  measurement that could produce a true positive.
+
 - [ ] 5.1 PHP/WordPress slice
   - Plugin and core weaknesses. `php2cpg` needs a PHP interpreter, which the image provides. **Validate on a
     sample that the model can read the pairs before vendoring** — the OWASP harvest found 238 and vendored 40
@@ -356,7 +370,16 @@
   - _Requirements: 9.1, 9.4, 9.5, 9.6, 9.7_
   - _Depends: 3.2_
 
-- [ ] 5.2 More vibe-code pairs
+- [ ] 5.2 More vibe-code pairs, and at least one as a whole repository
+  - The pair corpus has 70 vibe-py and 17 vfc-js pairs, all excerpts. Group 2 showed that excerpts cannot
+    see the defects a repository shows: nine engine bugs this session, none of them visible to the corpus,
+    the gates byte-identical through every fix.
+  - So the higher-value half of this task is a **pinned vibe-code repository**, not more pairs. It is the
+    cheapest measurement that could produce a second true-positive checkout, because Python and JavaScript
+    are the two languages whose fact tables are complete.
+  - Observable: a `benchmarks/repos/` recipe for a vibe-code checkout with in-scope known vulnerabilities,
+    and a measurement saying which were found at which rung.
+  - _Requirements: 9.1, 9.2, 4.1_
   - The slice where the stated audience works, currently 35 pairs. Same read-check discipline.
   - _Requirements: 9.3, 9.4, 9.5, 9.6, 9.7_
 
@@ -454,3 +477,65 @@ shapes were taught by the holdout pairs they then "recovered", turning a -0.059
     net-negative adoption is reverted automatically rather than argued about.
   - _Requirements: 8.2, 10.3_
   - _Depends: 7.4_
+
+## Group 8 — Per-repository state a developer owns
+
+> "an evolution (gepa, simba) style prompting with a declarative and persistent approach when evaluating
+> findings per target repo? so that devs can commit state and evolve the harness / dismiss findings"
+
+Yes, and Group 7 is half of it already -- 7.2's counter-example ledger is the persistent state, 7.3 is a model
+proposing from it, 7.4 is a split-respecting adoption. What this adds is the part that makes it a developer's
+rather than a maintainer's: the state lives **in the target repository**, committed beside the code, and it is
+declarative.
+
+**The boundary that decides the design.** A reflective optimiser may evolve a **prompt**, because a prompt is
+how we ask a question. It may not author **patterns or policy** -- the fact tables and the rulesets stay
+human-authored, and the tool never writes the declared policy file. So GEPA-style evolution has exactly one
+legitimate target here: the judge and residual questions in `model/pipeline.py`, which are already the
+LLM-facing surface. Evolving those against a repository's own dismissals is sound. Evolving sink tables from
+them is the thing this project has refused from the start, and the closed-loop leak is why it is right to.
+
+- [ ] 8.1 A dismissal file the developer owns
+  - `.openultrasast/dismissals.toml` in the TARGET repository: finding id or site, the rung it was dismissed
+    at, a reason, and who. Committed with the code, reviewed like code, and diffable -- a dismissal without a
+    reason is how a suppression file becomes a graveyard nobody can audit.
+  - The tool READS it and never writes it, which is the same rule the declared policy file already follows.
+    `ousast scan` reports dismissed findings as dismissed with their reason rather than hiding them, so a
+    stale dismissal is visible rather than silent.
+  - Observable: a scan of vampi with two dismissals reports 3 findings and 2 dismissed-with-reason, and the
+    file is untouched by the run.
+  - _Requirements: 5.1, 5.2_
+
+- [ ] 8.2 Dismissals are an evaluation set, per repository
+  - A dismissal is a labelled example: this site, at this rung, is not a finding here. Together with the
+    known vulnerabilities a `benchmarks/repos/` recipe already declares, that is a per-repository eval set
+    with both classes in it -- which is exactly what an optimiser needs and what a suppression list normally
+    throws away.
+  - Split it the way the pair corpus is split. A dismissal that teaches the prompt that recovers it is the
+    leak measured on 2026-09-05, where five of eleven candidate shapes were taught by the holdout pairs they
+    then recovered and a -0.059 Youden was reported as +0.059.
+  - Observable: the eval set builds from a recipe plus a dismissal file, with a declared train/holdout split
+    and a test that the holdout is never read during optimisation.
+  - _Requirements: 8.2, 10.2_
+  - _Depends: 8.1_
+
+- [ ] 8.3 Evolve the QUESTION, never the facts
+  - GEPA/SIMBA-style reflective evolution over `CANDIDATE_QUESTION` and `residual_question`, scored on 8.2's
+    train split: fewer dismissed sites affirmed, no known vulnerability lost.
+  - Hard boundaries, and they are the point of putting this in its own group: the optimiser proposes prompt
+    text only; it never edits a fact table, a ruleset or the declared policy file; an evolved prompt is
+    committed by a human like any other change; and the three gates stay byte-identical or the evolution is
+    a regression wearing a fix's clothes.
+  - Observable: an evolution record per accepted prompt -- train score before and after, holdout untouched,
+    known vulnerabilities still found, gates identical, and the diff of the prompt.
+  - _Requirements: 8.1, 8.2, 10.3_
+  - _Depends: 8.2, 7.4_
+
+- [ ] 8.4 Say what the harness learned, in the repository it learned it in
+  - Per-repository state is only useful if a developer can see what it did. A committed record of which
+    dismissals shaped the current prompt, and what that cost on the shared corpus, is what keeps this from
+    becoming a per-repo tuning that nobody can reason about.
+  - Observable: `ousast repos` or the report shows, for a target with evolved state, what was learned and
+    what it was measured against.
+  - _Requirements: 5.2, 10.4_
+  - _Depends: 8.3_

benchmarks/measurements/2026-09-09-report-proportionality.json

@@ -0,0 +1,24 @@
+{
+  "counted_never_dropped": "findings.json stays complete -- it is the record a baseline is compared against, and this is a view of it. The grouped section states the count, the rules, the top files, and that the model reaching none of them means no fact table covered the site rather than that the site is safe.",
+  "generated": "2026-09-09",
+  "measured": {
+    "libpng": {
+      "findings": 198,
+      "grouped": 161,
+      "report_lines": 679,
+      "sections_after": 39,
+      "sections_before": 198
+    },
+    "vampi": {
+      "findings": 60,
+      "grouped": 0,
+      "note": "its three pattern rows are under the limit and keep their sections",
+      "sections_after": 61,
+      "sections_before": 61
+    }
+  },
+  "rule": "A report is proportionate to what the engine can say. A finding whose only evidence is a text pattern -- evidence_level `static_corroboration` at rung `suspicion`, nothing corroborating the match -- is grouped by rule once that rule has more than three sites. Under three it keeps the sections it always had. Anything a proposer reasoned about keeps its section whatever its rung, because an undecided CLAIM about a site is not an observation that an API exists.",
+  "spec": "contributor-scan 2.11 (Req 5.1, 5.2)",
+  "what_it_does_not_fix": "vampi still emits 61 sections for a 520-line application: 38 model suspicions and 13 obligation findings, all reasoned claims and none of them collapsed by this rule. Whether fifty-one undecided claims is itself too many for a contributor is a separate question this task does not answer, and the honest position is that the rule fixes pattern noise rather than volume.",
+  "why_evidence_level_and_not_rung": "Filtering on rung was the obvious rule and it was wrong: the obligations checker emits at `suspicion` too, so hiding that rung would have hidden an entire detector. The discriminator is evidence_level. On libpng 163 of 198 rows are static_corroboration; on vampi 3 of 60 are. The model layer and the obligations checker carry `suspicion` there instead, which is the distinction the ladder was built to make."
+}

src/openultrasast/reports.py

@@ -28,11 +28,28 @@ def write_markdown_report(
     overlay_by_id = _overlay_by_id(overlay or [])
     cited: dict[str, Mapping[str, object]] = {}
     cited_obligations: dict[str, Mapping[str, object]] = {}
+    # THE RULE (contributor-scan 2.11): a report is proportionate to what the engine can say.
+    #
+    # A finding whose only evidence is a text pattern -- `evidence_level == "static_corroboration"`, nothing
+    # corroborating it beyond the match -- is grouped by rule instead of given a section each. libpng emitted
+    # 198 sections of which 163 were exactly that, asserting `memcpy` and `strcpy` are PRESENT in a library
+    # that uses them correctly throughout. That is the noise developers have learned to skip in every other
+    # tool, and shipping it costs the findings beside it their credibility.
+    #
+    # Grouped, never dropped, and proportionate rather than hidden. A rule with a handful of sites keeps its
+    # full sections -- one `python-flask-debug` is a finding a contributor can act on and must not be
+    # summarised away. A rule with fifty-four is a property of the codebase. Anything a proposer actually
+    # reasoned about -- the model layer, the obligations checker -- keeps its section whatever its rung,
+    # because an undecided CLAIM about a site is not an observation that an API exists.
+    patterned = _crowded_pattern_rules(findings)
+    crowded = {id(item) for item in patterned}
+    reasoned = [item for item in findings if id(item) not in crowded]
+
     lines = ["# OpenUltraSAST Report", "", f"Findings: {len(findings)}", "", "## Inventory", ""]
     if not findings:
         lines.append("No quick-mode findings were emitted.")
         lines.append("")
-    for finding in findings:
+    for finding in reasoned:
         verification = verification_by_id.get(finding.finding_id)
         disposition = overlay_by_id.get(finding.finding_id)
         lines.extend(
@@ -69,6 +86,8 @@ def write_markdown_report(
             cited_obligations[finding.finding_id] = info_o
             _append_obligation_lines(lines, info_o, (mechanisms or {}).get(str(info_o.get("known_fix") or "")))
         lines.extend(["", finding.rationale, ""])
+    if patterned:
+        _append_pattern_matches(lines, patterned)
     if cited:
         _append_mechanisms(lines, cited)
     if cited_obligations:
@@ -353,6 +372,64 @@ def _sarif_level(severity: str) -> str:
     return "note"
 
 
+# How many sites a pattern rule may have before it is summarised instead of listed in full. A rule at or
+# under this keeps the behaviour it always had; over it, the report stops repeating itself.
+_PATTERN_DETAIL_LIMIT = 3
+
+
+def _crowded_pattern_rules(findings: Sequence[StaticFinding]) -> list[StaticFinding]:
+    """Pattern-only findings belonging to a rule that matched more sites than are worth listing."""
+    counts: dict[str, int] = {}
+    for finding in findings:
+        if _is_pattern_only(finding):
+            rule = str(finding.finding_id).split(":", 1)[0]
+            counts[rule] = counts.get(rule, 0) + 1
+    return [
+        finding
+        for finding in findings
+        if _is_pattern_only(finding) and counts.get(str(finding.finding_id).split(":", 1)[0], 0) > _PATTERN_DETAIL_LIMIT
+    ]
+
+
+def _is_pattern_only(finding: StaticFinding) -> bool:
+    """Is a text match the only thing behind this finding?
+
+    `static_corroboration` is what the static layer emits when a pattern matched and nothing else spoke. A
+    proposer that reasoned about the site -- the model layer, the obligations checker -- carries `suspicion`
+    here even when its rung is also suspicion, and that is the distinction: an undecided CLAIM is not the
+    same as an observation that an API is present.
+    """
+    return str(finding.evidence_level) == "static_corroboration" and str(finding.rung) == "suspicion"
+
+
+def _append_pattern_matches(lines: list[str], patterned: Sequence[StaticFinding]) -> None:
+    """Pattern matches, grouped by rule. Proportionate: few sites are named, many are counted."""
+    by_rule: dict[str, list[StaticFinding]] = {}
+    for finding in patterned:
+        by_rule.setdefault(str(finding.finding_id).split(":", 1)[0], []).append(finding)
+
+    lines.extend(
+        [
+            "## Pattern matches",
+            "",
+            f"{len(patterned)} site(s) matched a static pattern with nothing corroborating them. A match says "
+            "an API is present, not that it is misused; the model reached none of these, and silence from the "
+            "model means no fact table covered the site rather than that it is safe. All of them are in "
+            "`findings.json`.",
+            "",
+        ]
+    )
+    for rule, items in sorted(by_rule.items(), key=lambda entry: (-len(entry[1]), entry[0])):
+        by_file: dict[str, int] = {}
+        for finding in items:
+            by_file[str(finding.path)] = by_file.get(str(finding.path), 0) + 1
+        top = sorted(by_file.items(), key=lambda entry: (-entry[1], entry[0]))[:3]
+        where = ", ".join(f"`{name}` ({count})" for name, count in top)
+        more = f", and {len(by_file) - len(top)} more file(s)" if len(by_file) > len(top) else ""
+        lines.append(f"- `{rule}` — {len(items)} site(s) across {len(by_file)} file(s): {where}{more}")
+    lines.append("")
+
+
 def _verification_by_id(verifications: list[VerificationResult]) -> dict[str, VerificationResult]:
     return {result.finding_id: result for result in verifications}
 

tests/test_reports.py

@@ -466,3 +466,54 @@ def _finding() -> StaticFinding:
         tags=["syscall_entry"],
         ranking_priority=3.0,
     )
+
+
+def _pattern_finding(rule: str, path: str, line: int) -> StaticFinding:
+    from dataclasses import replace
+
+    base = _finding()
+    return replace(base, finding_id=f"{rule}:{path}:{line}", path=path, line=line, title=f"{rule} needs review")
+
+
+def test_a_pattern_rule_with_many_sites_is_summarised_not_repeated(tmp_path: Path) -> None:
+    """contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting that `memcpy` and `strcpy`
+    are PRESENT in a library that uses them correctly throughout. That is the noise developers have learned
+    to skip in every other tool, and shipping it costs the findings beside it their credibility."""
+    crowded = [_pattern_finding("c-unsafe-copy", f"src/f{i}.c", i) for i in range(20)]
+    output = tmp_path / "report.md"
+
+    write_markdown_report(crowded, output)
+
+    text = output.read_text()
+    assert "## Pattern matches" in text
+    assert "20 site(s) across 20 file(s)" in text
+    assert text.count("## c-unsafe-copy needs review") == 0, "twenty sections for one rule is the noise"
+    assert "`findings.json`" in text, "counted, never dropped"
+
+
+def test_a_pattern_rule_with_few_sites_keeps_its_sections(tmp_path: Path) -> None:
+    """One `python-flask-debug` is a finding a contributor can act on, and must not be summarised away."""
+    output = tmp_path / "report.md"
+
+    write_markdown_report([_pattern_finding("python-flask-debug", "app.py", 17)], output)
+
+    text = output.read_text()
+    assert "## python-flask-debug needs review" in text
+    assert "## Pattern matches" not in text
+
+
+def test_a_reasoned_claim_keeps_its_section_whatever_its_rung(tmp_path: Path) -> None:
+    """An obligation or model finding sits at `suspicion` too, but it is a claim about a site rather than an
+    observation that an API exists. Twenty of them are twenty findings, not one property of the codebase."""
+    from dataclasses import replace
+
+    base = _finding()
+    reasoned = [
+        replace(base, finding_id=f"obligation:protected_read:api/u.py:{i}", evidence_level="suspicion", path="api/u.py", line=i)
+        for i in range(20)
+    ]
+    output = tmp_path / "report.md"
+
+    write_markdown_report(reasoned, output)
+
+    assert output.read_text().count(f"## {base.title}") == 20