marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
b8f8f727c651635b47a1f717dc275b983ecfd00d
Open_Ultra_SAST/benchmarks/measurements/2026-09-08-module-audit.json
T
Marius CiepluchandClaude Opus 5 b8f8f727c6 feat(model-grounded-detection): group 3, the evidence ladder and the judge 
3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The
existing evidence_level carries `static_corroboration`, emitted by the flat IR
overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as
model_corroborated would claim a CPG arbitrated something it never saw, and
rewriting every legacy finding to suspicion would be a product-visible
regression nobody asked for. So `rung` defaults to suspicion and is raised only
by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4:
nothing rises above suspicion without a Verdict behind it. The witness travels
with the rung, because a rung without its evidence is just a louder assertion.
It reaches markdown, JSON and SARIF.

3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive
outcomes: the model entails (report it, never call the model), the model
contradicts (drop the claim, record why), the model cannot decide (report
suspicion). The LLM gets one bounded typed question with tools=[] so it cannot
search for the site, asked only about the residual the graph cannot settle --
whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow
the CPG already found; it can never conjure one.

Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST
name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while
loop, or an import from the deleted round machinery. That test first matched
the word "votes" in the module's own docstring, which is why it now inspects
code rather than text.

Verified live against a real CPG, not only scripted rows:
  vuln   rung=model_entailed   request.args["name"] -> db.execute(...)
  fixed  rung=suspicion        no source reaches this sink
  model calls made: 0

Both sides arbitrated by the graph with the LLM never consulted -- the thing
the prior architecture could not do at any price.

678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
2026-09-08 13:49:55 +02:00

950 lines
26 KiB
JSON
Raw Blame History

{
"counts": {
"load_bearing": 80,
"standalone_capability": 7
},
"generated": "2026-09-08 (regenerated after group 3 added the ladder and the judge)",
"kept_with_a_lever_stripped": {
"improve/evolve.py + improve/validator.py": "the design's Removed list named these whole, but each is a mix: the mechanism lever (removed) and the deterministic rule-status improve loop that `ousast improve` still runs from benchmark evidence, which has no LLM in it and is not what this feature supersedes. Deleting the files would have taken a retained capability with them."
},
"modules": 87,
"note": "Classification of every source module after the group-1 subtraction. load_bearing = reachable from a gate, the CLI, or named by a surviving requirement; standalone_capability = its own entry point or protocol, deliberately off the scan path; orphaned = no importer, no entry point, no surviving requirement -- removed, never carried.",
"orphaned": [],
"removed_in_this_pass": {
"learning/ (16 modules)": "the noise architecture: rounds, acceptance, proposer, journal, canaries, verifiers, difficulty, scoring, classify, detectors, split, publish. Taxonomy, candidates and endpoint were re-homed to model/ first.",
"semantic/loo.py": "leave-one-out scoring of the mechanism store",
"semantic/mechanisms.py": "the mechanism store and the shape search that fed the optimisation loop",
"semantic/prove_budget.py": "embedding-ordered promotion budget, driven by the mechanism store",
"semantic/seed.py": "the mechanism exporter",
"semantic/variant_search.py": "variant search over the mechanism store",
"semantic/variants.py": "the shape-derivation optimisation surface; its guard vocabulary was ported verbatim to model/specs.py first, and trailing_name re-homed to semantic/ir.py",
"vectorstore.py": "a vector-store bakeoff harness with no production caller and no CLI entry, kept alive only by its own test -- the one genuine orphan the audit found"
},
"rows": [
{
"classification": "load_bearing",
"importers": [
"cli",
"gate",
"improve.evolve",
"map_gate",
"mcp",
"pairs"
],
"module": "benchmark",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "calibration",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"mcp"
],
"module": "cli",
"reason": "entry point"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"complexity.hints",
"complexity.ledger",
"complexity.map",
"map_gate",
"regress.candidate",
"semantic.prove_filter",
"tool_hunter"
],
"module": "complexity",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"complexity",
"complexity.map"
],
"module": "complexity.hints",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"complexity",
"complexity.map",
"regress.candidate"
],
"module": "complexity.ledger",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"complexity",
"complexity.hints",
"complexity.ledger",
"map_gate",
"regress.candidate",
"semantic.prove_filter",
"tool_hunter"
],
"module": "complexity.map",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"complexity",
"complexity.hints",
"complexity.map"
],
"module": "complexity.signals",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"harness",
"hunter_tools",
"model.endpoint",
"pairs",
"regress.candidate",
"regress.recipes",
"run"
],
"module": "config",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cpg.backend",
"model.judge",
"model.taint"
],
"module": "cpg",
"reason": "model-grounded-detection Req 4: the CPG seam package"
},
{
"classification": "load_bearing",
"importers": [
"model.judge",
"model.taint"
],
"module": "cpg.backend",
"reason": "model-grounded-detection Req 4.1/4.5: the single Joern subprocess boundary"
},
{
"classification": "load_bearing",
"importers": [
"cpg.backend"
],
"module": "cpg.capability",
"reason": "model-grounded-detection Req 4.2/4.3: the engine probe that degrades to suspicion"
},
{
"classification": "load_bearing",
"importers": [
"benchmark",
"calibration",
"cli",
"complexity.ledger",
"complexity.map",
"complexity.signals",
"fusion",
"gate",
"hunter",
"hunter_harness",
"hunter_tools",
"improve.evolve",
"map_gate",
"model.ladder",
"pairs",
"rank",
"regress.candidate",
"reports",
"scoring.project_score",
"semantic.obligations.check",
"semantic.overlay",
"semantic.prove_filter",
"stage_processors",
"tool_hunter",
"verification",
"verify_judge"
],
"module": "findings",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "standalone_capability",
"importers": [
"cli"
],
"module": "fusion",
"reason": "two-panel adjudication engine; reached from the scan's report stage and used standalone"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "gate",
"reason": "entry point"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"slot_contract"
],
"module": "harness",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "standalone_capability",
"importers": [
"cli",
"fusion",
"hunter_harness",
"stage_processors",
"verify_judge"
],
"module": "harness_ext",
"reason": "HarnessX extension surface; loaded by name when the optional harness is present"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"hunter_harness"
],
"module": "hunter",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "standalone_capability",
"importers": [
"cli"
],
"module": "hunter_harness",
"reason": "harness wiring for the tool hunter; used by the hunter path and by mcp"
},
{
"classification": "load_bearing",
"importers": [
"tool_hunter"
],
"module": "hunter_tools",
"reason": "imported by tool_hunter"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"improve.evolve"
],
"module": "improve",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"improve"
],
"module": "improve.evolve",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"improve",
"improve.evolve"
],
"module": "improve.journal",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"improve",
"improve.evolve"
],
"module": "improve.validator",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"skills"
],
"module": "index",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [],
"module": "map_gate",
"reason": "entry point"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"gate",
"hunter_tools",
"improve.evolve",
"map_gate",
"pairs",
"preprocess",
"stage_processors"
],
"module": "mapping",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "standalone_capability",
"importers": [
"cli"
],
"module": "mcp",
"reason": "narrow MCP server over stdio; its own `ousast mcp` entry point and OpenCode integration"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"model.candidates",
"model.judge",
"model.specs",
"model.taint"
],
"module": "model",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [],
"module": "model.audit",
"reason": "model-grounded-detection Req 3: this classifier; run by the maintainer, not by the scan"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "model.candidates",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [],
"module": "model.endpoint",
"reason": "model-grounded-detection Req 8: the judge client, wired by model/judge.py in group 3"
},
{
"classification": "load_bearing",
"importers": [],
"module": "model.judge",
"reason": "model-grounded-detection Req 8: one bounded question, checked against the model"
},
{
"classification": "load_bearing",
"importers": [
"model.judge",
"model.taint"
],
"module": "model.ladder",
"reason": "model-grounded-detection Req 5: the evidence ladder every finding carries"
},
{
"classification": "load_bearing",
"importers": [
"model.judge",
"model.taint"
],
"module": "model.specs",
"reason": "model-grounded-detection Req 7.2-7.4: the security vocabularies the CPG queries are parameterised by"
},
{
"classification": "load_bearing",
"importers": [
"model.judge"
],
"module": "model.taint",
"reason": "model-grounded-detection Req 6.1: taint reachability over the CPG, the flow-family arbiter"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"model.candidates",
"model.specs"
],
"module": "model.taxonomy",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "pair_gate",
"reason": "entry point"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"improve",
"improve.evolve",
"model.candidates",
"pair_gate"
],
"module": "pairs",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"findings",
"hunter",
"hunter_harness",
"improve.evolve",
"improve.validator",
"scoring.project_score",
"semantic.overlay",
"stage_processors"
],
"module": "policy",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"policy"
],
"module": "policy.verycode",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"complexity.hints",
"complexity.map",
"complexity.signals",
"findings",
"gate",
"hunter",
"hunter_harness",
"hunter_tools",
"improve.evolve",
"index",
"map_gate",
"mapping",
"model.candidates",
"pairs",
"rank",
"regress.candidate",
"semantic.overlay",
"skills",
"stage_processors"
],
"module": "preprocess",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "provenance",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"model.endpoint",
"tool_hunter"
],
"module": "provider",
"reason": "imported by model.endpoint, tool_hunter"
},
{
"classification": "load_bearing",
"importers": [
"model.endpoint",
"tool_hunter"
],
"module": "provider.openrouter",
"reason": "imported by model.endpoint, tool_hunter"
},
{
"classification": "load_bearing",
"importers": [
"calibration",
"cli",
"findings",
"gate",
"hunter",
"hunter_harness",
"hunter_tools",
"improve.evolve",
"map_gate",
"pairs",
"stage_processors"
],
"module": "rank",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"harness",
"model.judge",
"reports",
"semantic.obligations.intent"
],
"module": "redaction",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"regress.candidate"
],
"module": "regress",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"regress"
],
"module": "regress.candidate",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"regress",
"regress.candidate"
],
"module": "regress.recipes",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"regress",
"regress.candidate"
],
"module": "regress.safety",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"regress",
"regress.candidate"
],
"module": "regress.verdict",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "reports",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"findings",
"hunter",
"hunter_harness",
"improve.evolve",
"improve.validator",
"pairs",
"stage_processors"
],
"module": "ruleset",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"improve.validator",
"ruleset"
],
"module": "ruleset.store",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"reports"
],
"module": "run",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"regress.candidate",
"regress.recipes",
"regress.safety",
"regress.verdict"
],
"module": "sandbox",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"sandbox"
],
"module": "sandbox.probe",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"regress.candidate",
"regress.recipes",
"regress.safety",
"sandbox"
],
"module": "sandbox.runner",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"improve.evolve",
"stage_processors"
],
"module": "scoring",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"scoring"
],
"module": "scoring.project_score",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"cpg.capability",
"hunter_tools",
"model.candidates",
"model.specs",
"pairs",
"semantic.cst",
"semantic.engines",
"semantic.functions",
"semantic.ir",
"semantic.obligations.check",
"semantic.obligations.operations",
"semantic.obligations.shapes",
"semantic.overlay",
"semantic.prove_filter",
"semantic.taint"
],
"module": "semantic",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic.ir"
],
"module": "semantic.cst",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cpg.capability",
"semantic",
"semantic.overlay"
],
"module": "semantic.engines",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic",
"semantic.cst",
"semantic.engines"
],
"module": "semantic.extra",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"hunter_tools",
"model.candidates",
"model.specs",
"pairs",
"semantic",
"semantic.obligations.check",
"semantic.obligations.operations",
"semantic.obligations.shapes",
"semantic.overlay",
"semantic.taint"
],
"module": "semantic.facts",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"hunter_tools",
"pairs",
"semantic.obligations.shapes",
"semantic.overlay"
],
"module": "semantic.functions",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"hunter_tools",
"model.candidates",
"pairs",
"semantic.cst",
"semantic.functions",
"semantic.obligations.check",
"semantic.obligations.operations",
"semantic.obligations.shapes",
"semantic.overlay",
"semantic.taint"
],
"module": "semantic.ir",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"hunter_tools",
"model.candidates",
"model.specs",
"pairs",
"semantic.obligations.check",
"semantic.obligations.dominance",
"semantic.obligations.intent",
"semantic.obligations.operations",
"semantic.obligations.policy",
"semantic.obligations.shapes",
"semantic.obligations.siblings"
],
"module": "semantic.obligations",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"semantic.obligations",
"semantic.obligations.intent"
],
"module": "semantic.obligations.check",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"hunter_tools",
"pairs",
"semantic.obligations.check"
],
"module": "semantic.obligations.dominance",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"model.specs",
"pairs",
"semantic.obligations",
"semantic.obligations.check",
"semantic.obligations.operations",
"semantic.obligations.policy",
"semantic.obligations.shapes",
"semantic.obligations.siblings"
],
"module": "semantic.obligations.facts",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "semantic.obligations.intent",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"model.candidates",
"semantic.obligations.check",
"semantic.obligations.dominance",
"semantic.obligations.siblings"
],
"module": "semantic.obligations.operations",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli",
"semantic.obligations.check"
],
"module": "semantic.obligations.policy",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic.obligations",
"semantic.obligations.operations"
],
"module": "semantic.obligations.shapes",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic.obligations.check"
],
"module": "semantic.obligations.siblings",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic",
"semantic.prove_filter"
],
"module": "semantic.overlay",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic"
],
"module": "semantic.prove_filter",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"semantic.overlay"
],
"module": "semantic.taint",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "standalone_capability",
"importers": [
"hunter",
"hunter_harness"
],
"module": "skills",
"reason": "skill router; consumed by mcp and by the OpenCode integration, not by the scan path"
},
{
"classification": "standalone_capability",
"importers": [
"stage_processors"
],
"module": "slot_contract",
"reason": "the slot/contract protocol stage_processors implements"
},
{
"classification": "standalone_capability",
"importers": [],
"module": "stage_processors",
"reason": "HarnessX slot-contracted scan stages; pinned by the zero-dependency guard in test_gate"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "stages",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"model.endpoint"
],
"module": "tool_hunter",
"reason": "imported by model.endpoint"
},
{
"classification": "load_bearing",
"importers": [
"calibration",
"cli",
"fusion",
"reports",
"verify_judge"
],
"module": "verification",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
},
{
"classification": "load_bearing",
"importers": [
"cli"
],
"module": "verify_judge",
"reason": "reachable from gate, map_gate, pair_gate or the CLI"
}
],
"spec": "model-grounded-detection Req 3 (the outgrowth audit)",
"standalone": {
"fusion": "two-panel adjudication engine; reached from the scan's report stage and used standalone",
"harness_ext": "HarnessX extension surface; loaded by name when the optional harness is present",
"hunter_harness": "harness wiring for the tool hunter; used by the hunter path and by mcp",
"mcp": "narrow MCP server over stdio; its own `ousast mcp` entry point and OpenCode integration",
"skills": "skill router; consumed by mcp and by the OpenCode integration, not by the scan path",
"slot_contract": "the slot/contract protocol stage_processors implements",
"stage_processors": "HarnessX slot-contracted scan stages; pinned by the zero-dependency guard in test_gate"
}
}
Reference in New Issue View Git Blame Copy Permalink