mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 11:49:28 +00:00

Files

T

Marius Ciepluch 1ae41eea19 feat(propose-adjudicate-prove): overlay adjudicates inventory before prove

Regex inventory stays the proposer. MAP attaches promote/demote/unadjudicated/coverage
from intra-file taint (tree-sitter when a grammar parses, Python ast as last resort).
Deep sandboxes promotions only, ordered by the existing heuristic ranker plus optional
OpenRouter mechanism recall. The SAST pair slice scores overlay outcomes; the stage-1
detection gate and local pair gate stay inventory-only.

2026-09-04 13:57:33 +02:00

2.0 KiB

Raw Blame History

Vuln vs fixed pair corpus

This corpus measures harness efficiency, not cheat-sheet recall.

The stage-1 smoke gate (python -m openultrasast.gate) scans one tree that already contains the planted sinks. A pair eval instead scans two isolated trees:

vuln — only the vulnerable file. The harness should fire (true positive).
fixed — only the patched file, same relative path. The harness should stay silent on that CWE (true negative).

pair_correct = detected on vuln and silent on the fix. That is the signal the improvement loop consumes (miss on the vuln side, fp on the fix side).

Layout

Path	Role
`catalog.toml`	Runnable offline pairs (local fixtures + vendored GitHub functions)
`datasets.toml`	Pointers to public VFC datasets — do not vendor the dumps
`datasets/cwe-bench-java-project_info.csv`	120 vetted Java CVE buggy/fix commits
`github/`	Isolated function excerpts with provenance headers

Pairs in this directory must not join LANGUAGE_MANIFESTS.

Slices

local — in-tree fixture counterparts (app.py vs safe.py, …). CI gate: every local pair must pass (python -m openultrasast.pair_gate).
github — real commit/function pairs (SVEN, CWE-Bench-Java, NVD patterns). Measured, not a 90% smoke gate. Misses here are the honesty / evolve-the-harness dashboard.
sast — OWASP Benchmark Java/Python true-vs-false files and Juliet bad-vs-good sinks (benchmarks/pairs/sast/). Score is Youden (TPR − FPR). Honesty dashboard, not a CI fail.

Commands

uv run ousast pairs
uv run ousast pairs --slice github --json
uv run python -m openultrasast.pair_gate

Growing the corpus

Prefer function-level before/after from a reviewed source (SVEN jsonl, a CWE-Bench-Java fix_info.csv row, a GHSA with a tiny patch). Isolate the changed function into vuln / fixed files with the same relpath. Leave whole-repo clones to --fetch later; CI stays offline.

2.0 KiB Raw Blame History Unescape Escape