measure(contributor-scan): first ontology coverage run, and it found a family we do not have

NodeGoat models 18 of 175 distinct call names, covering 12 percent of 1,185 calls. That share is not
a score: most of what a program calls is not security-relevant, and a high number would mean the
tables were too loose. It is a baseline to move deliberately.

The top product candidate is res.render, 23 calls with five in a method that reads untrusted input.
Task 14.3 reasoned its way to that same gap by hand and declined to model it as a flow sink; the
instrument found it in one run, which is the validation that matters. It also surfaced Math.random in
the user data-access module, and weak randomness is a weakness class this ontology has no family for
at all -- so no family census could have found it, because the family list is itself the blind spot.
That is the kind of gap only a coverage instrument finds.

It had to be taught what is not an API. The first run ranked _tmp_3.push and __ecma.Array.factory
first, which is how the frontend lowers a destructuring and an array literal, and a Cypress suite and
a Gruntfile held four of the next five. Both are separated now, because noise at the top of a
worklist is how a worklist stops being read.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

benchmarks/measurements/2026-09-21-ontology-coverage-nodegoat.json

@@ -0,0 +1,45 @@
+{
+  "schema_version": 1,
+  "experiment": "ontology-coverage-first-run-v1",
+  "date": "2026-09-21",
+  "task": "contributor-scan 7.1",
+  "claim_scope": "What one subject calls that the ontology does not model. A candidate list, not an ontology.",
+  "subject": "nodegoat",
+  "language": "javascript",
+  "coverage": {
+    "distinct_call_names": 175,
+    "calls_total": 1185,
+    "modelled_names": 18,
+    "modelled_calls": 143,
+    "modelled_call_share": 0.121
+  },
+  "ontology_size": {
+    "discharger": 40,
+    "operation": 36,
+    "sanitizer": 10,
+    "setting": 10,
+    "sink:injection": 5,
+    "sink:output_encoding": 12,
+    "sink:path": 6,
+    "sink:prototype": 7,
+    "sink:untrusted_destination": 15,
+    "source": 11
+  },
+  "reading": "18 of 175 distinct call names are modelled, covering 12% of the calls. That number is not a score: most of what any program calls is not security-relevant, and a high share would mean the tables were too loose. It is a baseline to move deliberately.",
+  "leads_it_found": {
+    "res.render": "23 calls, five in a method that reads untrusted input, and the highest-ranked product candidate. Task 14.3 reasoned its way to this same gap by hand and declined to model it as a flow sink because template escaping is a property of the engine's configuration. The instrument found it in one run, which is the validation that matters.",
+    "Math.random": "7 calls in app/data/user-dao.js. Weak randomness is a weakness class this ontology has no family for at all, so no census could have surfaced it: the family list itself is the blind spot. This is the kind of gap only a coverage instrument finds.",
+    "parseInt": "9 calls, one near a source. A coercion, and the JavaScript sanitizer list has no coercion entry while Python's has five.",
+    "app.post and app.use": "route registration, near untrusted input, and the place middleware guards are attached",
+    "db.collection": "the accessor whose result carries the operations the access-control family models"
+  },
+  "noise_it_had_to_be_taught_to_drop": {
+    "synthesised_names": "`_tmp_3.push` and `__ecma.Array.factory` were the two most frequent candidates in the first run. They are how the frontend lowers a destructuring and an array literal, not names the code calls.",
+    "tests_and_build_scripts": "A Cypress suite and a Gruntfile held four of the five most frequent remaining candidates. Ranked together with product code they would have buried the two real leads."
+  },
+  "limits": [
+    "The `file` and `line` on a candidate are its first occurrence, which misleads when one name is called from several files; the distinct-spelling count is reported beside it.",
+    "One subject, one language. The instrument says nothing about whether a candidate is security-relevant; that judgement is the point of requiring provenance before an entry.",
+    "Coverage share counts calls, not findings. A name modelled but never reached contributes to the share and nothing else."
+  ]
+}

benchmarks/push/ontology_gap.py

@@ -21,9 +21,14 @@ from typing import Any
 
 from openultrasast.cpg.backend import JoernBackend
 from openultrasast.model.contracts import ExecutionBudget
+from openultrasast.model.layout import is_test_path
 from openultrasast.model.specs import config_specs, dominance_specs, taint_specs
 from openultrasast.preprocess import enumerate_source_files
 
+# Build tooling a repository runs but does not ship. Named here rather than guessed from a path, because a
+# file called Gruntfile is a build script wherever it sits.
+_BUILD_SCRIPTS = frozenset({"Gruntfile.js", "gulpfile.js", "webpack.config.js", "rollup.config.js", "karma.conf.js"})
+
 
 def modelled_tokens(language: str) -> dict[str, tuple[str, ...]]:
     """Every token any fact table names for this language, by the role it plays.
@@ -117,7 +122,17 @@ def main() -> int:
     }
     record["modelled_seen"] = [{k: r[k] for k in ("name", "count") if k in r} for r in known[: args.top]]
     wanted = ("name", "spelling", "count", "nearSource", "file", "line")
-    record["candidates"] = [{k: r[k] for k in wanted if k in r} for r in unknown[: args.top]]
+
+    # A test helper and a build script are not product API. The layout facts already know which paths those
+    # are, and on the first measured subject a Gruntfile and a Cypress suite held four of the five most
+    # frequent candidates -- ranked first, they would have buried the two real leads beneath them.
+    def product(row: dict[str, Any]) -> bool:
+        where = str(row.get("file") or "")
+        return bool(where) and not is_test_path(where) and Path(where).name not in _BUILD_SCRIPTS
+
+    ranked = [{k: r[k] for k in wanted if k in r} for r in unknown]
+    record["candidates"] = [r for r in ranked if product(r)][: args.top]
+    record["candidates_in_test_or_build"] = [r for r in ranked if not product(r)][:20]
     record["candidates_near_a_source"] = [r for r in record["candidates"] if int(r.get("nearSource", 0)) > 0][:20]
     save()
     cleanup = getattr(graph, "cleanup", None)