Maintainer correction: execution isn't possible for all SAST targets, and the
bugs this project owns -- IDOR, missing authz, injection -- don't crash
anything. Execution is the easy arbiter only for memory-safety-in-C, which is
not our world.
What execution provided was an independent, deterministic arbiter of whether a
claim is real -- not the same LLM that made it, not a noisy vote. That arbiter
can be built statically, from the code, as a program model the claim is checked
against. LLM proposes; the model disposes. Same architecture, same repair of
the root cause, no runnable target required.
The evidence ladder becomes static and deterministic:
suspicion -> model_corroborated (claim consistent with resolved dataflow /
a guard its siblings carry is absent) -> model_entailed (the model alone
establishes it) -> execution_confirmed (deferred tier, C memory-safety only).
model_corroborated and above are deterministic -- run the checker twice, same
verdict -- which removes the run-to-run noise the same way a crash would, with
no K-runs and no target to build. Our IR candidate enumerator (96.6% ceiling),
binding chains and obligation checker stop being prompt context and become the
arbiter itself.
The course: promote those assets from inputs to arbiter, delete the noise
layer, make the LLM a proposer the model audits, keep the corpus as the
model's calibration set (the fix diff is what the model should see), and defer
execution to Clearwing for the memory-safety classes only. Build the arbiter we
can build from code; adopt the one we cannot, when a class demands it.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Written after five evolve rounds produced zero decisions distinguishable from
noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals.
The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to
average (K runs), floor (noise budget), gate (sign test) and slowly improve
(evolve loop). 120 runs produced 120 distinct trajectories -- the variance is
the loop's freedom to search, not the model's uncertainty. And the premise is
empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day
in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the
2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The
lever is architecture, not prompt optimization.
What the winners share, and Clearwing is the open instance of: the LLM is a
judgement layer on an execution substrate, and truth comes from the sandbox --
a crash, a sanitizer report, a patch that stops the crash -- not from a score
against a labeled corpus. Clearwing's poc_runner (apply diff, recompile,
re-run, does-the-crash-still-happen) replaces our entire noise architecture. We
built K-runs because we had no ground truth; they generate their own.
The course: adopt Clearwing as the substrate (the maintainer already forks it),
delete the noise architecture rather than fix it, and contribute the one thing
Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator
(12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker
reach the dangerous web bugs that never crash a sandbox, which Clearwing's
crash-oracle model cannot. The open research problem is a differential oracle
for those classes: the vulnerable-parent/fix-commit pair as ground truth, with
a request the fixed version rejects and the vulnerable one serves.
Recommendation is explicitly not to rebuild: eighteen defects this session,
all ours, against a 19k-line battle-tested MIT substrate. Not approved.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Design approved in session. Every one of the 36 criteria is claimed by a task.
Group 1 calls no model at all. The candidate enumerator, the ceiling report
and the stop-or-go reading are offline, so the condition that ends this
feature -- a deciding slice whose ceiling sits below its recorded detection
rate -- is reachable before a cent is spent. The ruleset variant of this same
design would have failed that condition at 12.1% against 64.9%.
Group 2 is the only work that touches scoring.py, and its observable is that
nothing changes: the recorded round-zero artifacts are the pin, and if they
cannot be reproduced exactly then the refactor is wrong, not the numbers.
Group 3 builds the detector, and task 3.3 is the test that would have caught
the defect this spec exists for -- two runs must make an identical sequence of
model calls. It lives in the suite and runs repeatedly, not in a one-off
measurement.
Group 4 is the maintainer's emphasis as code: deciding slices, development
slices, the cross-slice rejection by name, and the overfitting gap printed
beside every headline number.
Group 5 compares the two detectors per slice, re-measures the trajectory count
and the noise floor, and names -- without amending -- the learning-harness
requirements that a collapsed floor would retire.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Requirements approved in session. All 36 criteria trace to a component.
The shape: a deterministic pass enumerates candidate sites from the semantic
IR, shaped by family -- call sites for injection and path, operations with
their guards for access control, bindings and literals for configuration. The
model answers one bounded typed question per candidate with no tools and no
choice of what to look at. The predicates that decide whether a finding counts
are applied while it is being made, with one retry carrying the violation, and
they come from a single definition that the scorer also uses, so the two
cannot drift.
Determinism is a design property, not a hope: candidates are ordered by
(path, line, name) and batched at a fixed size, so two runs make an identical
sequence of model calls. That is the direct answer to 120 runs producing 120
distinct trajectories.
Phase 1 is the candidate ceiling, offline, no model call, measured on the
real-project slices as well as the teaching one. If the ceiling on a deciding
slice comes in below that slice's current detection rate, the design stops
there having spent nothing. Building that in as the first phase is deliberate:
the ruleset variant of this design would have capped recall at 12.1% against a
64.9% baseline, and only a measurement caught it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Renamed from detector-optimizer: the brief's own conclusion is that the
optimizer is not the next thing to build, and a spec should be named for what
it delivers.
Seven requirements, 37 criteria. The load-bearing ones:
Req 1 -- candidates come from the semantic IR, never the pattern ruleset,
and the enumerator's ceiling is published as the limit on recall. Measured
before building: ruleset 12.1%, IR 96.6%.
Req 2 -- one bounded typed question per candidate, no tools, no choice of
what to look at, and a test asserts two runs make an identical sequence of
model calls.
Req 3 is the maintainer's emphasis and it decides acceptance. The slices
harvested from real projects decide; hand-written vibe-py is a development
slice that cannot on its own justify a change; a change that improves vibe-py
and regresses a deciding slice is rejected by name; and the overfitting gap
-- 83.3% against 44.4% at baseline -- is published beside every headline
number. The candidate ceiling is measured separately on real-world slices,
because their functions are larger and the enumerator's behaviour there is
not implied by its behaviour on teaching code.
Req 5 -- the K-run and noise-floor machinery is re-measured against the
constrained detector and deleted if the floor collapses, naming the
learning-harness requirements that would be amended.
Not approved; design not started.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
