mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
86f434ab897312babce4dcd1cc57c427845b63f6
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marius CiepluchandClaude Opus 5
|
f1c341df6c |
feat(model-grounded-detection): group 5, corpus calibration and honest reporting
5.1 inverts what the corpus is for. Previously a pair the detector missed was a
DETECTOR failure, to be averaged away over K runs and chased with a prompt
change. The arbiter is deterministic: it will miss that pair the same way every
time, and the only thing that moves it is better modelling. So a pair the model
cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every
gap carries one even when the caller supplies none -- a bare gap teaches
nothing.
`covered` means the model reached a strictly higher rung on the vulnerable side
than on its fix. Flagging both sides equally is not coverage: it distinguishes
nothing, which is exactly what group 2 measured taint reachability doing on
every parameterised SQL pair. The harvest seam takes the fix commit and derives
the vulnerable side as its parent, with the fix diff as the oracle -- nothing
hand-labelled.
5.2 keeps three disciplines from the deleted publish layer, each because its
absence caused a real error here:
* per-slice rows always -- `render` RAISES without them. A mixed headline
once hid agent-written code scoring 44% against the development slice's 83%.
* the overfitting gap beside the headline, and ABSENT rather than zero when a
slice is missing: reading an unmeasured slice as "no overfitting" is the
flattering failure, not a neutral one.
* the rung on every finding, so a suspicion cannot be read as a result.
Run end to end against the committed ceiling artifact, the report says things
worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an
overfitting gap of +16%, and gaps grouped by family with their reasons --
injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no
verdict), untrusted_destination 5 (4 entailed on BOTH sides).
713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules.
A full re-measure with the closure-scoping fix is in flight; these numbers are
the pre-fix baseline and will move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
d9ceeb6a09 |
feat(model-grounded-detection): 4.2 constant abstraction, the third arbiter form
A configuration bug has no flow to follow and no guard to dominate.
CORS(app, origins="*") is dangerous because of the value it is SET TO, so the
arbiter evaluates the setting's arguments against the closed permissive set in
the obligation facts -- the third form beside taint reachability and guard
dominance, and the one Req 6.3 names.
ENTAILED a security setting is passed a literal in the permissive set
CORROBORATED the argument is computed, so constant abstraction cannot decide
it -- that residual is the judge's, never silently "safe"
(none) every literal is outside the set, or the call is not a setting
Two Joern node-shape bugs, both found by running it and both of the same class
as group 2's receiver bug -- the graph's indexing is not what the source text
suggests:
* a keyword argument (`origins="*"`) is an assignment-shaped node, so
isLiteral is false on the argument itself and the literal sits in its
subtree; literals are collected from the argument's AST now.
* Joern indexes the receiver at 0, positional arguments from 1, and NAMED
arguments at -1. Settings are configured overwhelmingly by keyword, so
argumentIndexGt(0) -- correct when counting arity for the taint shape test
-- dropped exactly the arguments that matter and every keyword-configured
setting read as "computed".
Live, deterministic, zero model calls:
create_app -> model_entailed CORS(app, origins = "*"): permissive literal "*"
safe_app -> no finding (origins = "https://app.example")
All three arbiter forms now work against a real CPG: taint reachability for the
flow families, guard dominance for absence, constant abstraction for config.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
fec67253f3 |
feat(model-grounded-detection): 4.1 guard dominance, 4.2 the missing sink facts
4.1 is the arbiter for the class no crash oracle reaches. An access-control bug
is the ABSENCE of a guard: no flow to follow, no crash to reproduce. What the
model can establish is a consistency violation -- an obligated operation no
discharging guard governs, in a file where its siblings are governed by one.
"Governed" turned out not to be one relation, and asking pure CFG dominance of
the operation gets two of three shapes wrong. Measured against the real engine:
identity constraint the guard is INSIDE the operation's arguments
filter_by(owner_id = current_user.id)
identity branch abort(403).controlledBy == [note.owner_id != current_user.id]
-- the check runs AFTER the fetch, so it never dominates
the operation, but it controls whether the value escapes
dominating guard a discharger-naming call that dominates the operation
That is why the query uses controlledBy rather than dominatedBy for shape 2.
Two bugs found by running it, both the same class as the sanitizer bug in group
2 -- part of a fact flattened, the part carrying the signal dropped:
* dominance_specs used only `calls` and `decorators`, omitting
identity_sources (current_user, request.user) and constraint_params
(owner_id, user_id) -- which is how an identity constraint is actually
written. The arbiter reported "no guard" on handlers that plainly had one.
request_sources and permissive_values stay excluded: they describe the
vulnerable shape, and including them would let the arbiter read a bug as
its own fix.
* matching operations by code containment yielded four or five desugared rows
per operation (the bare `tmp1.filter_by` receiver among them, carrying none
of the guards the real call has), so a guarded operation could be flagged.
Matching is by call name now.
Live, deterministic, zero model calls:
leaky -> model_entailed no discharging guard, while 2 sibling handlers are guarded
edit_note -> no finding (ownership check controls the denial)
list_notes -> no finding (identity constraint in the query)
4.2 authors the sink facts the go/no-go measured as its largest miss class (11
of 23): Django ORM raw SQL, and the four families that previously had no fact
routing to them by CWE at all -- path, output_encoding, untrusted_destination,
prototype. Each entry is a documented dangerous API of the named library with
its provenance recorded inline, and rule_ids left empty because a sink fact is
not a detection rule. Flagged for maintainer review as data, given the standing
"no LLM-authored patterns" constraint. The exact-set assertion written in task
1.1 to hold this gap open is updated rather than deleted, so it still fails if a
family ever loses its seed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
|
||
|
Marius CiepluchandClaude Opus 5
|
b8f8f727c6 |
feat(model-grounded-detection): group 3, the evidence ladder and the judge
3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The existing evidence_level carries `static_corroboration`, emitted by the flat IR overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as model_corroborated would claim a CPG arbitrated something it never saw, and rewriting every legacy finding to suspicion would be a product-visible regression nobody asked for. So `rung` defaults to suspicion and is raised only by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4: nothing rises above suspicion without a Verdict behind it. The witness travels with the rung, because a rung without its evidence is just a louder assertion. It reaches markdown, JSON and SARIF. 3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive outcomes: the model entails (report it, never call the model), the model contradicts (drop the claim, record why), the model cannot decide (report suspicion). The LLM gets one bounded typed question with tools=[] so it cannot search for the site, asked only about the residual the graph cannot settle -- whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow the CPG already found; it can never conjure one. Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while loop, or an import from the deleted round machinery. That test first matched the word "votes" in the module's own docstring, which is why it now inspects code rather than text. Verified live against a real CPG, not only scripted rows: vuln rung=model_entailed request.args["name"] -> db.execute(...) fixed rung=suspicion no source reaches this sink model calls made: 0 Both sides arbitrated by the graph with the LLM never consulted -- the thing the prior architecture could not do at any price. 678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
2b973dc0db |
feat(model-grounded-detection): tasks 2.1-2.2, the Joern seam and the taint verdict
The whole engine integration is one subprocess boundary: joern-parse builds a cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between fence markers, because Joern's stdout carries a banner and a REPL prompt around whatever a script emits. Nothing is imported in-process, so the core install keeps dependencies = [] -- a test walks the seam's AST to assert it imports only the standard library and names no JVM binding. Fail closed everywhere. A build or query that errors, times out, or prints something unparseable yields no verdict and the caller reports suspicion. The taint verdict distinguishes this from absence deliberately: a query returning None means "the engine could not decide", which is not "there is no flow", and conflating them is how a missing engine silently becomes a clean bill of health. The verdict is where the ladder earns its keep. An unsanitized source->sink path is model_entailed -- the graph establishes it and the LLM's claim is no longer load-bearing. A path a sanitizer may break is model_corroborated: the flow is real, but whether the sanitizer suffices is the semantic residual the LLM answers, checked against this verdict rather than averaged over runs. The chosen flow is picked by a total order (shortest, then lexicographic), so two runs over one CPG cannot disagree -- Req 6.4 asserted directly. The query itself is the part our flat IR could not do: reachableByFlows walks the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is exactly the one-hop limit that held measured entailment at 2.2%. Capability detection reuses semantic/engines.joern_available rather than growing a second notion of "installed". The audit manifest is regenerated; the new modules are named by Req 4/5/6 rather than counted as orphans. Task 2.3, the go/no-go measurement, needs Joern installed and has not run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |
||
|
Marius CiepluchandClaude Opus 5
|
9ed0e45c5f |
feat(model-grounded-detection): group 1, subtract the noise architecture
99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16 modules, ~4,100 lines) and six semantic/ modules removed, with the three gates byte-identical to 2026-09-06-gate-baseline.txt at every step. The sequence is what made it safe: vocabularies ported first (1.1), importers detached second (1.2-1.3), deletion third (1.4). Every import break during the deletion was therefore a detachment miss in the importer and was fixed there -- no module was restored to satisfy one. Both misses were the same shape and were fixed as a class: operations.py and shapes.py each imported trailing_name from the deleted variants.py, so it moved to semantic/ir.py, its natural home, and both importers changed in one edit. Ported before deleting: the closed guard vocabulary (verbatim, nothing else holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from the retained ruleset facts rather than copying them. Re-homed: families -> model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint, plus Region, labeled_family and labeled_spans. labeled_family no longer falls back to the LLM classifier -- an unlabeled pair is honestly unknown. Four deviations, each recorded in the spec notes with its reason. improve/ evolve.py and validator.py were stripped, not deleted: each mixes the mechanism lever (gone) with the deterministic rule-status loop `ousast improve` still runs, which has no LLM in it; a test now asserts the lever's absence by symbol so it cannot creep back. The learning CLI group became `model candidates`, because deleting it outright would have orphaned the retained enumerator by removing its only entry point. thinking moved from [learning] to [models]. A gap named rather than papered over: no retained sink fact routes by CWE to path, output_encoding, untrusted_destination or prototype -- the same hole the entailment ceiling diagnosed. The test asserts the seeded set and the gap set exactly, so task 4.2 must update it when it authors those sinks. The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness with no production caller and no CLI, alive only through its own test. The five named subsystems each carry an explicit decision. Manifest committed, with a test asserting the tree still matches it. Group 2 (the Joern go/no-go) has not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG |