mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
chore(p0): retire regress, the skill router, the module audit tool and vocabulary.sc
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts). Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep now records a regress_retired degradation and equals standard; proof is the search-with-proof arm's job), the sandbox docker runner and probe that only regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers only regress called; skills.py and index.py with the hunter pool's skill routing; model/audit.py; cpg/queries/vocabulary.sc with benchmarks/push/ontology_gap.py. Docs and the scan skill describe the retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed, 28 skipped; 3,331 tests collect. Co-Authored-By: Claude Fable 5.1 <[email protected]>
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
94e96c6885
commit
f5a904b7f2
@@ -1,101 +0,0 @@
|
||||
// What this repository calls that the ontology does not model (contributor-scan 7.1).
|
||||
//
|
||||
// Five of seven families established nothing in the 2026-09 family census, and every one of them failed for
|
||||
// an ontology reason with the flow engine ready and nothing to aim at: no HTTP client for a server-side
|
||||
// request forgery, no collection read for an object-reference defect, no WordPress spelling for a
|
||||
// deserialization. Each was found by investigating one family at a time. This answers the question directly.
|
||||
//
|
||||
// It needs no dataflow. The structural pass already walks every call, which is why a census that took three
|
||||
// investigations can be one query.
|
||||
//
|
||||
// What it reports is a CANDIDATE LIST, never an ontology. A name here becomes a fact only when someone
|
||||
// resolves it to a library API, states the weakness class and records provenance. Adding a name because a
|
||||
// subject called it is how a general-purpose tool becomes a fixture of its own benchmark.
|
||||
//
|
||||
// Parameters: cpgFile, modelled (comma-separated tokens any fact table names), sources (comma-separated
|
||||
// source patterns, used only to say whether a call sits in a method that reads untrusted input).
|
||||
// Output: fenced JSON array of {name, spelling, count, modelled, nearSource, file, line}.
|
||||
@main def exec(cpgFile: String, modelled: String = "", sources: String = "") = {
|
||||
importCpg(cpgFile)
|
||||
|
||||
def split(raw: String): List[String] = raw.split(",").map(_.trim).filter(_.nonEmpty).toList
|
||||
|
||||
val modelledTokens = split(modelled)
|
||||
val sourcePatterns = split(sources)
|
||||
|
||||
def boundedPattern(t: String) =
|
||||
java.util.regex.Pattern.compile("(?<![A-Za-z0-9_$])" + java.util.regex.Pattern.quote(t) + "(?![A-Za-z0-9_$])")
|
||||
|
||||
val patterns = modelledTokens.map(t => (t, boundedPattern(t)))
|
||||
|
||||
// The callee, which is the only part of a call that names what is being called. Matching a whole node's
|
||||
// text is the defect this project has now fixed six times.
|
||||
def calleeText(code: String): String = {
|
||||
val open = code.indexOf('(')
|
||||
val head = if (open < 0) code else code.substring(0, open)
|
||||
head.trim.takeRight(120)
|
||||
}
|
||||
|
||||
// Names the FRONTEND synthesised are not names the code calls. jssrc2cpg lowers a destructuring to
|
||||
// `_tmp_3.push` and an array literal to `__ecma.Array.factory`, and on the first measured subject those
|
||||
// were the two most frequent candidates in the list -- noise at the top of a worklist is how a worklist
|
||||
// stops being read.
|
||||
def synthesised(name: String): Boolean =
|
||||
name.startsWith("<") || name.startsWith("__ecma") || name.matches("^_tmp_?[0-9]*\\..*") || name.startsWith("_tmp")
|
||||
|
||||
val calls = cpg.call.filterNot(c => c.name.startsWith("<operator") || synthesised(c.name)).l
|
||||
|
||||
// Memoised per method: does this method read something the ontology calls untrusted input? A gap that sits
|
||||
// beside a source is worth more attention than one that does not, and that is a ranking hint rather than a
|
||||
// claim about any flow.
|
||||
val sourceMemo = scala.collection.mutable.Map.empty[String, Boolean]
|
||||
def nearSource(c: io.shiftleft.codepropertygraph.generated.nodes.Call): Boolean =
|
||||
if (sourcePatterns.isEmpty) false
|
||||
// CPS-IT/mailqueue Extension::KEY has a null .method; it cannot be near a method source.
|
||||
else Option(c.method).exists { m =>
|
||||
sourceMemo.getOrElseUpdate(
|
||||
m.fullName,
|
||||
m.ast.isCall.exists(x => sourcePatterns.exists(p => boundedPattern(p).matcher(x.code).find()))
|
||||
)
|
||||
}
|
||||
|
||||
def isModelled(c: io.shiftleft.codepropertygraph.generated.nodes.Call): Boolean = {
|
||||
val callee = calleeText(c.code)
|
||||
patterns.exists { case (token, pattern) => c.name == token || pattern.matcher(callee).find() }
|
||||
}
|
||||
|
||||
case class Row(name: String, spelling: String, modelledHere: Boolean, near: Boolean, file: String, line: Int)
|
||||
|
||||
val rows = calls.map { c =>
|
||||
Row(
|
||||
c.name,
|
||||
calleeText(c.code),
|
||||
isModelled(c),
|
||||
nearSource(c),
|
||||
c.file.name.l.headOption.getOrElse(""),
|
||||
c.lineNumber.getOrElse(-1)
|
||||
)
|
||||
}
|
||||
|
||||
val grouped = rows.groupBy(_.name).toList.map { case (name, items) =>
|
||||
val first = items.head
|
||||
ujson.Obj(
|
||||
"name" -> name,
|
||||
// The commonest qualified spelling, because `get` and `needle.get` are different leads.
|
||||
"spelling" -> items.groupBy(_.spelling).toList.maxBy(_._2.size)._1,
|
||||
"count" -> items.size,
|
||||
"modelled" -> items.exists(_.modelledHere),
|
||||
"nearSource" -> items.count(_.near),
|
||||
"file" -> first.file,
|
||||
"spellings" -> items.map(_.spelling).distinct.size,
|
||||
"line" -> first.line
|
||||
)
|
||||
}
|
||||
|
||||
// Unmodelled first and by frequency, because that ordering IS the worklist.
|
||||
val ordered = grouped.sortBy(o => (o("modelled").bool, -o("count").num.toInt))
|
||||
|
||||
println("---OUSAST-CPG-BEGIN---")
|
||||
println(ujson.write(ujson.Arr(ordered: _*)))
|
||||
println("---OUSAST-CPG-END---")
|
||||
}
|
||||
Reference in New Issue
Block a user