From 0beae46ba9568b96e09ef6a0372873d608a0f36f Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 21 Sep 2026 17:12:24 +0200 Subject: [PATCH] feat(contributor-scan): ask a repository what it calls that the ontology does not model Five of seven families established nothing in the family census, and every one failed for an ontology reason with the flow engine ready and nothing to aim at. Each was found by investigating one family at a time over days. vocabulary.sc asks it once: census the calls, subtract everything any fact table names, rank the remainder by how often it is called and whether it sits in a method that reads untrusted input. It needs no dataflow, because the structural pass already walks every call. The harness reads its token set from the SPECS rather than from the TOML, so it measures what the analyser actually consults: a fact the loader drops is not part of the ontology however neatly it is written. Ontology sizes today are 152 tokens for JavaScript, 167 for Python, 84 for PHP. Both carry the constraint the maintainer set: the output is a candidate list, never an ontology. A name becomes a fact only when someone resolves it to a library API, states the weakness class and records provenance, because adding a name because a subject called it is how a general-purpose tool becomes a fixture of its own benchmark. Also records what the bounded PHP transfer attempt found. At 25 regions the first case now runs to completion and writes a full artifact in 1,706 s, where at 100 it had exhausted a 2,400 s deadline with nothing. But 25 of 4,461 regions is 0.6 percent of the population and the declared target was not among them, so the question was never asked. Task 13.3 therefore depends on 7.3, skipping arbitration where the evidence says there is nothing to find: raising the budget is already measured to fail, and making most of the budget unnecessary has not been tried. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- .kiro/specs/pre-push-safety-net/tasks.md | 5 +- benchmarks/push/ontology_gap.py | 132 ++++++++++++++++++++ src/openultrasast/cpg/queries/vocabulary.sc | 91 ++++++++++++++ 3 files changed, 227 insertions(+), 1 deletion(-) create mode 100644 benchmarks/push/ontology_gap.py create mode 100644 src/openultrasast/cpg/queries/vocabulary.sc diff --git a/.kiro/specs/pre-push-safety-net/tasks.md b/.kiro/specs/pre-push-safety-net/tasks.md index 9eca1c9..35554cd 100644 --- a/.kiro/specs/pre-push-safety-net/tasks.md +++ b/.kiro/specs/pre-push-safety-net/tasks.md @@ -558,8 +558,11 @@ Preserve all completed task evidence below. - **Named by the 2026-09-20 correction.** The not-established record came from two instrument defects, not from the analyser: the census verdict checked question completion before findings, and a 500-region budget asks 2,526 questions, which no ceiling this project sets can hold. Bounded to 25 and 100 regions, PMPro completes with no timeout and establishes a true-positive SQL injection at `adminpages/discountcodes.php:749`. - Replay the three declared comparisons with the corrected harness at a region budget that completes, and record the declared target's evidence or its absence. State the budget as a limit of the run, not as a property of the repository: a bounded population establishes transfer, never coverage. - Done when all three comparisons complete with their declared target evidence recorded, or the budget they need is stated as a finding. + - _Attempted 2026-09-21, and the attempt named its own blocker._ At 25 regions the first case now runs to completion and writes a full artifact, in 1,706 s, where at 100 it had exhausted a 2,400 s deadline with nothing. But 25 regions of 4,461 is 0.6% of the population, and the declared target at `classes/class.memberorder.php:936` was not among them: zero findings at the target, because the question was never asked. A bounded population establishes transfer only if it CONTAINS the target, and this one cannot. + - **So this task depends on contributor-scan 7.3, not on a larger budget.** The population that holds the target is affordable only if arbitration stops being paid for questions whose evidence says there is nothing to find. Raising the budget is the thing already measured to fail; making most of the budget unnecessary is the thing that has not been tried. + - _Two further observations from that run, recorded because they cost nothing to keep._ The head scan cost 1,178 s against the base's 219 s, which is worth explaining before M2 designs around either number. And every coverage reason was `ambiguous_line_correspondence`, one per file — so in the PUSH path, unlike the census, correspondence is what stands between this subject and a completed question. The partition work removed one blocker and revealed the next. - _Boundary: Evaluation harness, full-run integration_ - - _Depends: 13.2_ + - _Depends: 13.2, contributor-scan 7.3_ - _Requirements: 6.2, 6.3, 8.1, 8.2_ - [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds - **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan. diff --git a/benchmarks/push/ontology_gap.py b/benchmarks/push/ontology_gap.py new file mode 100644 index 0000000..e903394 --- /dev/null +++ b/benchmarks/push/ontology_gap.py @@ -0,0 +1,132 @@ +"""What a repository calls that the ontology does not model (contributor-scan 7.1). + +Five of seven families established nothing in the 2026-09 family census, and every one of them failed for an +ontology reason rather than an analysis one: the flow engine was ready and had nothing to aim at. Each gap +was found by investigating one family at a time, over days. This asks the question once. + +What it produces is a CANDIDATE LIST. A name here becomes a fact only when someone resolves it to a library +API, states the weakness class and records provenance -- the ontology describes libraries, frameworks and +language constructs, never repositories, and adding a name because a subject called it is how a +general-purpose tool becomes a fixture of its own benchmark. +""" + +from __future__ import annotations + +import argparse +import collections +import json +import time +from pathlib import Path +from typing import Any + +from openultrasast.cpg.backend import JoernBackend +from openultrasast.model.contracts import ExecutionBudget +from openultrasast.model.specs import config_specs, dominance_specs, taint_specs +from openultrasast.preprocess import enumerate_source_files + + +def modelled_tokens(language: str) -> dict[str, tuple[str, ...]]: + """Every token any fact table names for this language, by the role it plays. + + Read from the specs rather than from the TOML, so the instrument measures what the analyser actually + consults: a fact the loader drops is not part of the ontology however neatly it is written. + """ + roles: dict[str, set[str]] = collections.defaultdict(set) + for family, spec in taint_specs(language=language).items(): + roles[f"sink:{family}"].update(spec.sinks) + roles["source"].update(spec.sources) + roles["sanitizer"].update(spec.sanitizers) + roles["sanitizer"].update(spec.safe_shape_sinks) + for spec in dominance_specs(language=language).values(): + roles["operation"].update(spec.operations) + roles["discharger"].update(spec.dischargers) + for spec in config_specs(language=language).values(): + roles["setting"].update(spec.settings) + return {role: tuple(sorted(tokens)) for role, tokens in roles.items()} + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--out", type=Path, required=True) + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--name", required=True) + parser.add_argument("--language", required=True) + parser.add_argument("--deadline", type=float, default=1800.0) + parser.add_argument("--top", type=int, default=60, help="how many unmodelled candidates to record") + args = parser.parse_args() + args.out.mkdir(parents=True, exist_ok=True) + + roles = modelled_tokens(args.language) + every = sorted({token for tokens in roles.values() for token in tokens}) + record: dict[str, Any] = { + "schema_version": 1, + "experiment": "ontology-coverage-v1", + "subject": args.name, + "language": args.language, + "root": str(args.root), + "claim_scope": ( + "A candidate list, not an ontology. A name here is a lead to a library API; it becomes a fact " + "only with a weakness class and provenance." + ), + "ontology_size": {role: len(tokens) for role, tokens in sorted(roles.items())}, + } + target = args.out / f"{args.name}-{args.language}.json" + + def save() -> None: + target.write_text(json.dumps(record, indent=2, default=str) + "\n") + + files = enumerate_source_files(args.root) + record["inputs"] = {"files": len(files), "bytes": sum(p.stat().st_size for p in files if p.is_file())} + save() + if not files: + record["error"] = "no readable source" + save() + return 2 + + started = time.monotonic() + budget = ExecutionBudget(started + args.deadline, 5.0) + backend = JoernBackend(execution_budget=budget) + graph = backend.build(args.root, language=args.language, execution_budget=budget) + if graph is None: + record["error"] = f"graph build failed: {backend.last_failure}" + save() + return 3 + + rows = backend.query( + graph.cpg_path, + "vocabulary", + {"modelled": ",".join(every), "sources": ",".join(roles.get("source", ()))}, + ) + record["seconds"] = round(time.monotonic() - started, 1) + if not isinstance(rows, list): + record["error"] = "the vocabulary query did not answer" + save() + return 4 + + known = [r for r in rows if isinstance(r, dict) and r.get("modelled")] + unknown = [r for r in rows if isinstance(r, dict) and not r.get("modelled")] + calls_total = sum(int(r.get("count", 0)) for r in rows if isinstance(r, dict)) + record["coverage"] = { + "distinct_call_names": len(rows), + "calls_total": calls_total, + "modelled_names": len(known), + "modelled_calls": sum(int(r.get("count", 0)) for r in known), + # The share of CALLS, not of names: one name called four hundred times matters more than four + # hundred called once, and a name count would flatter the ontology. + "modelled_call_share": round(sum(int(r.get("count", 0)) for r in known) / calls_total, 3) if calls_total else 0.0, + } + record["modelled_seen"] = [{k: r[k] for k in ("name", "count") if k in r} for r in known[: args.top]] + wanted = ("name", "spelling", "count", "nearSource", "file", "line") + record["candidates"] = [{k: r[k] for k in wanted if k in r} for r in unknown[: args.top]] + record["candidates_near_a_source"] = [r for r in record["candidates"] if int(r.get("nearSource", 0)) > 0][:20] + save() + cleanup = getattr(graph, "cleanup", None) + if callable(cleanup): + cleanup() + backend.close_session() + print(json.dumps({"subject": args.name, "language": args.language, "coverage": record["coverage"]}), flush=True) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/openultrasast/cpg/queries/vocabulary.sc b/src/openultrasast/cpg/queries/vocabulary.sc new file mode 100644 index 0000000..3285361 --- /dev/null +++ b/src/openultrasast/cpg/queries/vocabulary.sc @@ -0,0 +1,91 @@ +// What this repository calls that the ontology does not model (contributor-scan 7.1). +// +// Five of seven families established nothing in the 2026-09 family census, and every one of them failed for +// an ontology reason with the flow engine ready and nothing to aim at: no HTTP client for a server-side +// request forgery, no collection read for an object-reference defect, no WordPress spelling for a +// deserialization. Each was found by investigating one family at a time. This answers the question directly. +// +// It needs no dataflow. The structural pass already walks every call, which is why a census that took three +// investigations can be one query. +// +// What it reports is a CANDIDATE LIST, never an ontology. A name here becomes a fact only when someone +// resolves it to a library API, states the weakness class and records provenance. Adding a name because a +// subject called it is how a general-purpose tool becomes a fixture of its own benchmark. +// +// Parameters: cpgFile, modelled (comma-separated tokens any fact table names), sources (comma-separated +// source patterns, used only to say whether a call sits in a method that reads untrusted input). +// Output: fenced JSON array of {name, spelling, count, modelled, nearSource, file, line}. +@main def exec(cpgFile: String, modelled: String = "", sources: String = "") = { + importCpg(cpgFile) + + def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList + + val modelledTokens = split(modelled) + val sourcePatterns = split(sources) + + def boundedPattern(t: String) = + java.util.regex.Pattern.compile("(? (t, boundedPattern(t))) + + // The callee, which is the only part of a call that names what is being called. Matching a whole node's + // text is the defect this project has now fixed six times. + def calleeText(code: String): String = { + val open = code.indexOf('(') + val head = if (open < 0) code else code.substring(0, open) + head.trim.takeRight(120) + } + + val calls = cpg.call.filterNot(_.name.startsWith(" sourcePatterns.exists(p => x.code.contains(p))) + ) + + def isModelled(c: io.shiftleft.codepropertygraph.generated.nodes.Call): Boolean = { + val callee = calleeText(c.code) + patterns.exists { case (token, pattern) => c.name == token || pattern.matcher(callee).find() } + } + + case class Row(name: String, spelling: String, modelledHere: Boolean, near: Boolean, file: String, line: Int) + + val rows = calls.map { c => + Row( + c.name, + calleeText(c.code), + isModelled(c), + nearSource(c), + c.file.name.l.headOption.getOrElse(""), + c.lineNumber.getOrElse(-1) + ) + } + + val grouped = rows.groupBy(_.name).toList.map { case (name, items) => + val first = items.head + ujson.Obj( + "name" -> name, + // The commonest qualified spelling, because `get` and `needle.get` are different leads. + "spelling" -> items.groupBy(_.spelling).toList.maxBy(_._2.size)._1, + "count" -> items.size, + "modelled" -> items.exists(_.modelledHere), + "nearSource" -> items.count(_.near), + "file" -> first.file, + "line" -> first.line + ) + } + + // Unmodelled first and by frequency, because that ordering IS the worklist. + val ordered = grouped.sortBy(o => (o("modelled").bool, -o("count").num.toInt)) + + println("---OUSAST-CPG-BEGIN---") + println(ujson.write(ujson.Arr(ordered: _*))) + println("---OUSAST-CPG-END---") +}