marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 19:59:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
253 Commits 1 Branch 11 Tags
af7e33ef3e7c511c29fd4f5c98fe9e2dbd11791a
Commit Graph
4 Commits
Author SHA1 Message Date
Marius CiepluchandClaude Opus 5 177c7f3719 feat(model): first repository measurement, and the four things it found 
contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than
as excerpts, with benchmarks/repos/measure.py recording build time, query
time, arbitration time, peak memory, cost, unjudged regions, and whether
each known vulnerability was found and at which rung.

The corpus could not have found any of this. Every defect below needs a
repository to become visible, and three of them were invisible behind a
passing test suite.

**The batch path was unreachable.** query_batch was implemented, tested and
committed while CpgResult exposed only `run`, so the driver's capability
check found nothing and fell back to one JVM per region-family: the first
attempt at this measurement ran eighteen minutes on 520 lines, issuing
`--param function=` one at a time. Every test called query_batch directly,
so none of them noticed. CpgResult now carries run_batch and a test asserts
the capability is reachable through the object the driver is handed.
The earlier 275s->106s improvement attributed to batching came from region
deduplication alone.

**The budget stopped free work.** It bounds model calls, but was enforced
by breaking the region loop -- spending a paid limit to stop unpaid CPG
arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the
graph entails for nothing, was never reached. It now withholds the judge
only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent
because unaffordable is not silent because decided), and the BOLA is
entailed by the model layer.

**A region with no function asked about the whole repository.** Sending
function="" matched every hit in the tree and the driver attributed that
answer to whoever asked: one `host='0.0.0.0'` in app.py became eight
identical entailed findings in eight files that do not contain it. Queries
now carry the region's file. Entailed findings 14 -> 7, query time
70.9s -> 40.9s. The single-request form stays unscoped so the committed
pair measurements remain reproducible.

**The JVM had no heap bound.** Joern claimed a quarter of physical RAM and
runs were killed under memory pressure at different points, so the numbers
described the host rather than the engine. -J-Xmx2048m by default,
OPENULTRASAST_CPG_HEAP_MB to raise it.

Also: arbitrated findings carried `function:line` where the report layer
read `path:line`, so every entailed finding reached a contributor with a
function name where its file belonged; and the witness line was parsed by
splitting on a comma, which produced sites like
`config.py:17): permissive literal '0.0.0.0':?`.

Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end,
869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the
only shape: VAmPI's SQL injection runs from a handler parameter into a
function in another module, and a per-function region sees neither end.
That is the ordinary shape of a real injection bug, so it is recorded as
task 2.4 along with three smaller defects this run exposed.

Gates byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
 2026-09-09 08:46:21 +02:00
Marius CiepluchandClaude Opus 5 c35426e186 perf(model): one Joern invocation per query kind, not one per region 
JVM startup dominated a scan: a call per region per family put a ten-line
file at 275s over three regions, and would have put a thousand regions at
roughly fifty hours. The queries themselves are milliseconds once the CPG
is loaded, so the cost was almost entirely process starts.

scan_repository() is now collect -> batch -> arbitrate. The three .sc
scripts accept a `requests` JSON map and answer all of them in one run,
keeping the single-request form so the committed measurements stay
reproducible byte-for-byte. Arbiters are untouched: each is handed a
CpgResult whose `run` returns its own prefetched rows.

The batcher builds parameters by calling request_params() extracted from
taint.py, dominance.py and config_value.py rather than reconstructing
them. A second copy that drifts from the arbiter's is the exact shape of
the five family-assumption bugs the predecessor spec spent a session
finding; there is now one definition per family and the batcher borrows
it.

Regions are also deduplicated by (path, function), collapsing the
nameless entry-point rows that were producing `model:injection:?:10`.

Measured on the same ten-line file: 275s -> 106s, three duplicate
findings -> one `model:injection:ping:10`, rung unchanged at
model_entailed. Gates byte-identical.

Server mode is recorded as task 2.3, deliberately sequenced after the
repository measurement: whether the remaining ~100s floor is worth a
process model, a weakened subprocess seam and a Scala-evaluating HTTP
endpoint depends on a build-time-to-query-time ratio only 2.2 produces.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
 2026-09-09 07:57:35 +02:00
Marius CiepluchandClaude Opus 5 fe45081b43 fix(ci): run ruff format, and record run 5 at 90% pair-correct 
CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every
change this session and never once ran `ruff format --check`, though CI runs
both. Lint passing is not formatting passing, and I treated one as evidence for
the other for the whole session.

Formatting applied. The full CI sequence now reproduces locally: format, lint,
mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the
semantic job's 16 marked tests.

Also records run 5 of the pipeline measurement, which the vampi fix lifted as
predicted:

                       hunter (K=5)   pipeline (K=1)
  pair-correct              76.7%          90.0%
  leak                      12.3%           6.7%
  detection                    --          96.7%
  cost                     ~$0.69        $0.0068

27 of 30 pairs. Five runs, each step from a measurement contradicting an
assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which
exposed the discarded verdict) -> 80.0 (residual question matched to its
arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer
contradicting its own evidence).

One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default
rather than a security-setting call. Out of reach before the settings work and
still is; contorting the arbiter around one pair would be worse than the miss.

The caveat is recorded beside the number: 30 pairs of one slice, and the slice
iterated against all session. The +16% overfitting gap against vfc-js is why
contributor-scan Req 8.1/8.2 exist.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
 2026-09-08 19:46:58 +02:00
Marius CiepluchandClaude Opus 5 2b973dc0db feat(model-grounded-detection): tasks 2.1-2.2, the Joern seam and the taint verdict 
The whole engine integration is one subprocess boundary: joern-parse builds a
cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between
fence markers, because Joern's stdout carries a banner and a REPL prompt around
whatever a script emits. Nothing is imported in-process, so the core install
keeps dependencies = [] -- a test walks the seam's AST to assert it imports
only the standard library and names no JVM binding.

Fail closed everywhere. A build or query that errors, times out, or prints
something unparseable yields no verdict and the caller reports suspicion. The
taint verdict distinguishes this from absence deliberately: a query returning
None means "the engine could not decide", which is not "there is no flow", and
conflating them is how a missing engine silently becomes a clean bill of health.

The verdict is where the ladder earns its keep. An unsanitized source->sink
path is model_entailed -- the graph establishes it and the LLM's claim is no
longer load-bearing. A path a sanitizer may break is model_corroborated: the
flow is real, but whether the sanitizer suffices is the semantic residual the
LLM answers, checked against this verdict rather than averaged over runs. The
chosen flow is picked by a total order (shortest, then lexicographic), so two
runs over one CPG cannot disagree -- Req 6.4 asserted directly.

The query itself is the part our flat IR could not do: reachableByFlows walks
the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is
exactly the one-hop limit that held measured entailment at 2.2%.

Capability detection reuses semantic/engines.joern_available rather than
growing a second notion of "installed". The audit manifest is regenerated; the
new modules are named by Req 4/5/6 rather than counted as orphans.

Task 2.3, the go/no-go measurement, needs Joern installed and has not run.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
 2026-09-08 12:23:40 +02:00