From 7308570ba66bfc12e2593cf90fbc1ad580d1f8b0 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Thu, 10 Sep 2026 12:59:24 +0200 Subject: [PATCH] docs(contributor-scan): 5.12 -- the ranker knows one framework, everything else is "unknown" The rank SCALE is generic: `_ACCESS_RANK` maps abstract access levels to numbers and knows no framework. The EVIDENCE feeding it does not -- `_wordpress_hook_access` holds the literals `wp_ajax_nopriv_` and `wp_ajax_`, `_rest_route_access` holds `__return_true`. One framework, written in Python. The consequence is not "slightly worse elsewhere", it is no ordering at all. Anything unrecognised falls to review-required = 0.30 and ties break alphabetically by path. That is where CVE-2023-23488's region sits -- position ~1036 of 4,463, below every REST handler -- so a 500-region budget never reaches it. On a Laravel or Django codebase EVERY region lands in that tier: thousands of regions ordered by filename. Three layers, and the model is only the third: * facts, not code -- an [[access]] fact declares hook patterns and their access level, the way [[dispatch]] already replaced hardcoded add_action/apply_filters today; * generic features needing no framework knowledge -- modelled sinks in the file, a modelled source present, call-graph distance to the nearest sink, shipped, an unsanitized path to any sink. This is what would move CVE-2023-23488's region on its own, and it should exist BEFORE any model, or the model's contribution cannot be measured against anything; * the LLM for the residue the first two cannot separate. Ranking is a safe place for a model because it cannot manufacture a finding: it decides what is looked at, never what is reported. The arbiters still decide, the gates stay byte-identical, and a misrank costs recall -- now visibly, since regions_truncated reports 500 of 4,463. Records the constraints that must be settled before spending model budget: batched over a prefilter (4,463 calls a scan is not a CI tool), leave-one-repository-out (this project already has the train-on-test scar), a recorded ranking so baselines diff, and honest degradation with no model. A model proposing FACTS for an unknown framework is a different, slower loop needing human approval -- ranking at runtime is not policy, a fact table is. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/contributor-scan/tasks.md | 54 +++++++++++++++++++++++++++ src/openultrasast/cpg/backend.py | 15 +++++++- 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/.kiro/specs/contributor-scan/tasks.md b/.kiro/specs/contributor-scan/tasks.md index 99d5b81..8b6d4d8 100644 --- a/.kiro/specs/contributor-scan/tasks.md +++ b/.kiro/specs/contributor-scan/tasks.md @@ -805,6 +805,60 @@ - _Requirements: 4.4, 6.1, 8.1, 9.1_ - _Depends: 5.8_ +- [ ] 5.12 The ranker knows one framework, and everything else lands in "unknown" + - > "the ranker hardcodes wp nodes etc. which means it's not generalizeable. I'd like the LLM to be a + > flexible layer here to avoid hardcoded / overly specific ranker implementations" + - **What is actually hardcoded, precisely.** The rank SCALE is generic -- `_ACCESS_RANK` maps abstract + access levels to numbers and knows nothing about any framework. The EVIDENCE feeding it does not: + `mapping._wordpress_hook_access` is a Python function containing the literals `wp_ajax_nopriv_` and + `wp_ajax_`, and `_rest_route_access` contains `__return_true`. One framework, written in code. + - **The consequence is not "slightly worse on other frameworks", it is no ordering at all.** Anything the + classifier does not recognise falls to `review-required` = 0.30, and within a tie the order is + alphabetical by path -- arbitrary. On Paid Memberships Pro that is where CVE-2023-23488's region sits, + below every REST handler at 0.80, at position ~1036 of 4,463. With a 500-region budget the arbiter never + sees it. A Laravel, Symfony or Django codebase would have EVERY region in that tier: 4,463 regions + ordered by filename. + - The fix is three layers, and the LLM is only the third: + - **Facts, not code.** Framework vocabulary moves into the semantic tables the way `[[dispatch]]` already + did today. An `[[access]]` fact declares hook-name patterns and their access level, so + `wp_ajax_nopriv_*` is a row rather than an `if`. Covers every framework somebody has written facts for, + costs nothing at runtime, and is auditable. + - **Generic features that need no framework knowledge at all.** Every region can be scored on things the + CPG and the existing fact tables already know: how many modelled sinks its file contains, whether a + modelled source appears in it, its call-graph distance to the nearest sink, whether it is shipped, + whether any sink call in it lacks a sanitizer on any path. A method holding an unprepared `$wpdb` + query one hop from a parameter should outrank an empty getter WITHOUT anyone having written a + WordPress rule. This is the layer that would have moved CVE-2023-23488's region on its own, and it + should be built before any model is involved -- otherwise the LLM's contribution cannot be measured + against anything. + - **The LLM for the residue.** Regions the facts cannot classify and the features cannot separate. It + sees the feature vector plus a short excerpt and orders them. + - **Why ranking is a safe place for a model, and adjudication is not.** A ranker cannot manufacture a + finding. It decides what is LOOKED AT, never what is reported: the arbiters still entail or stay silent, + the three gates stay byte-identical, and a misranked region costs recall rather than producing a false + claim. The failure is also now visible rather than silent, because `regions_truncated` reports "500 of + 4,463 examined" -- an unexamined region can no longer be mistaken for a clean one. + - Constraints that belong in the design before any model budget is spent: + - **Cost.** One call per region is 4,463 calls a scan and nobody runs that in CI. The model ranks a + prefiltered candidate set in batches, or it is not shipped. + - **Leak.** Four pinned CVEs is a tiny training set and this project has the scar already -- see the + closed-loop train-on-test finding, where the improve lever learned from holdout pairs and inflated its + own numbers. Optimisation must be leave-one-repository-out and the reported figure must come from a + plugin the optimiser never saw. + - **Reproducibility.** Two runs of one repository must agree, or a baseline diff means nothing. The + ranking is recorded in the manifest so a change in what was examined is auditable rather than + invisible. + - **Degradation.** With no model configured the scan still runs on layers one and two, and the report + says which ranker produced the order. + - A model could also PROPOSE facts for an unrecognised framework -- "this codebase registers handlers with + `Router::get(...)`" as a candidate `[[dispatch]]` row. That is a different and slower loop, and it must + go through human approval, because a fact is policy and the standing rule is that no policy is + LLM-authored. Ranking at runtime is not policy; a fact table is. + - Observable: the rank position of each pinned CVE's region, before and after, on a repository the + optimiser did not see. `getMemberOrderByCode` is at ~1036 of 4,463 today. + - _Requirements: 4.4, 9.1_ + - _Depends: 5.8_ + ## Group 6 — Regression baselines - [ ] 6.1 Repository baseline and delta diff --git a/src/openultrasast/cpg/backend.py b/src/openultrasast/cpg/backend.py index e1fd292..c1bf50c 100644 --- a/src/openultrasast/cpg/backend.py +++ b/src/openultrasast/cpg/backend.py @@ -40,7 +40,12 @@ BEGIN = "---OUSAST-CPG-BEGIN---" END = "---OUSAST-CPG-END---" BUILD_TIMEOUT_SECONDS = 900 # a large tree takes minutes; the go/no-go records what it actually cost +# A slice's questions answer in seconds; a repository's do not. Measured on a 637-file WordPress plugin: +# 13 dominance requests answer in 50s (most of it loading the graph) and 2,500 taint requests exceed 300s. +# The constant was chosen when a "scan" meant an excerpt, and a repository scan is not a slower slice -- it +# is a different order of question, so the ceiling is configurable rather than a number to keep raising. QUERY_TIMEOUT_SECONDS = 300 +QUERY_TIMEOUT_ENV = "OPENULTRASAST_CPG_QUERY_TIMEOUT" # A JVM with no -Xmx takes a quarter of physical RAM for its heap. On a contributor's laptop that is the # difference between a scan running in the background and a scan the machine notices, and it made the # first repository measurements unreproducible: runs were killed under memory pressure at different @@ -210,7 +215,7 @@ class JoernBackend: runner: Runner | None = None build_timeout: int = BUILD_TIMEOUT_SECONDS - query_timeout: int = QUERY_TIMEOUT_SECONDS + query_timeout: int = field(default_factory=lambda: _configured_timeout()) heap_mb: int = 0 # 0 means read the environment, then fall back to CPG_HEAP_MB queries_dir: Path = field(default_factory=lambda: QUERIES_DIR) # Why the last build refused, so the driver can name it instead of reporting a bare `cpg_build_failed`. @@ -712,6 +717,14 @@ def _hoist_shared(rendered: dict[str, dict[str, str]]) -> dict[str, str]: return shared +def _configured_timeout() -> int: + """The query ceiling, from the environment or the default. Never zero or negative.""" + raw = os.environ.get(QUERY_TIMEOUT_ENV, "").strip() + if raw.isdigit() and int(raw) > 0: + return int(raw) + return QUERY_TIMEOUT_SECONDS + + def _render(value: object) -> str: """CPGQL parameters are strings; a sequence becomes a comma-separated list the script splits.""" if isinstance(value, str):