Open_Ultra_SAST/.kiro/specs/execution-grounded-detection/brief.md

Corrective brief: execution-grounded detection

Status: course-setting. Supersedes learning-harness and constrained-detector. Not a requirements document — a decision about what to build, what to delete, and what to adopt, written after the learning-harness line produced numbers its own measurements say to discard.

1. The premise we got wrong

OpenUltraSAST was built on an unstated assumption: that an LLM is a noisy oracle whose judgement must be averaged (K runs), floored (per-family noise budget), gated (sign test), and slowly improved (evolve loop). Every mechanism in learning-harness follows from that assumption. This session measured all of it and it does not hold:

• Five evolve rounds, $2.17, zero decisions distinguishable from chance — because the acceptance rule compares two ensemble votes over a detector that never runs the same investigation twice.
• 120 runs of one detector produced 120 distinct trajectories. The variance is the loop's freedom to search, not the model's uncertainty about an answer.
• The noise floor a regression budget is derived from measures how often an unconstrained search happens to reach the sink — so the worse the search wanders, the more collateral damage a round is licensed to do.

The assumption is also empirically false as a claim about LLMs, and the evidence is now overwhelming:

• Google Big Sleep (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software — a stack buffer underflow in SQLite — found by variant analysis, a sandboxed test case, and an auto-generated root-cause report. Not pattern-matching; reproduction.
• DARPA AIxCC finals (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code, patched 43 of 54 synthetic vulnerabilities and uncovered 18 previously-unknown real 0-days. Buttercup (Trail of Bits, 2nd place) found 28 vulnerabilities across 20 CWEs at 90% accuracy, $181 per point, using non-reasoning LLMs only, ~$152 per task, 45 minutes each.

The lever in every one of those systems is architecture, not prompt optimization. Buttercup won with non-reasoning models. Nobody evolved a prompt with a genetic loop. The question "can LLMs find security bugs" was answered yes by 2024; we spent this project answering a question the field had closed, with the wrong architecture.

2. What the successful systems share, and Clearwing is the open one

Big Sleep, Atlantis (AIxCC 1st), Buttercup (2nd), and Clearwing (/home/mc/Source/clearwing, Lazarus-AI, MIT, an open reproduction of Anthropic's Glasswing) share one shape:

The LLM is a judgement layer on top of a deterministic + execution substrate. Truth comes from the sandbox — a crash, a sanitizer report, a patch that stops the crash — not from a score against a labeled corpus.

Clearwing's evidence ladder is the whole thesis in one type:

suspicion → static_corroboration → crash_reproduced → root_cause_explained
          → exploit_demonstrated → patch_validated

The load-bearing gate is poc_runner.py: apply a candidate diff inside the sandbox, recompile, re-run the PoC, and return a boolean for does the crash still happen. That single primitive replaces our entire noise architecture. You do not average five noisy runs when a container can tell you whether the bug is real. We built K-runs and noise floors because we had no ground truth. They generate their own.

3. What Clearwing does better, mapped to what failed here

Our failure this session	What Clearwing does instead
K-runs / noise floor / sign test / regression budget — a statistical treatment for a structural problem	Execution is the arbiter. poc_runner reproduces or it doesn't. stability.py runs the PoC across N fresh containers (≥90% = stable) to kill flakiness at the source instead of averaging it.
Evolve loop: 5 rounds, 0 reliable decisions, $2.17 to learn nothing	A fixed specialist roster dispatched by file tags — memory_safety, kernel_syscall, crypto_primitive, web_framework, logic_auth, general — plus variant analysis (verify a bug, generate a grep+semantic pattern, find its structural twins, iterate to fixpoint). Density compounds within a run, no optimizer.
Second judge / fusion, gesturing at independence	4-axis adversarial validator (REAL / TRIGGERABLE / IMPACTFUL / GENERAL) that sees only the finding, PoC and exploit — never the discovery transcript — steel-mans both sides, and must attempt reproduction.
Corpus: 12 hand-harvested function excerpts per family, scored by text-inside-function, capped by twins/straddles/truncation defects we spent days repairing	Corpus is real repos at vulnerable-parent commits + OSS-Fuzz (7000 targets in full mode), run blind, with the fix commit used only as an after-the-fact oracle. Truth is generated by the sandbox, not annotated by us. CampaignConfig scales this to hundreds of repos with shared budget, checkpoint/resume, and diminishing-returns detection.
Detector chooses what to look at (555 grep calls vs 71 entry-point lookups)	Deterministic preprocessing feeds candidates: callgraph → intra-procedural taint (source→sink) → 3-axis ranker → hunter. The model judges; it does not search for the sink. This is exactly the constrained-detector design we had started — Clearwing is the mature version of it.

4. The one thing OpenUltraSAST proved that Clearwing lacks

This is not a total loss, and the course is not "throw everything away." Two results here are real, measured, and absent from Clearwing:

1. IR-driven candidate enumeration with a published ceiling. We measured that the pattern ruleset caps recall at 12.1% while call-sites-and-bindings from the semantic IR reach 96.6% across the web slices, a median of ten sites per function. Clearwing's candidate feeder is C-first (taint tables for memcpy-shaped bugs); its web_framework specialist is a single unconstrained prompt with no equivalent candidate enumerator for logic/web classes.

2. Absence-bug detection. Our obligations work — an access-control bug is the absence of a guard around an operation its siblings protect — has no analogue in Clearwing, whose ladder is built around a crash. An IDOR or a missing authorization check never crashes a sandbox. Clearwing's execution ground-truth model does not reach these classes at all, and they are exactly the "dangerous but low-hanging" web bugs this project was chartered to catch.

So the gap Clearwing leaves open is the web/logic taxonomy without a crash oracle — which is where our IR candidate enumerator, our family taxonomy, and our obligation checker are genuinely ahead.

5. The course

Adopt Clearwing as the substrate; contribute the web/logic detection path it lacks; delete our noise architecture entirely.

Concretely, superseding both prior specs:

1. Delete, do not fix. The K-run machinery, the per-family noise floor, the regression budget, the sign test, the evolve loop, the acceptance rule, the mechanism-lever optimizer. All of it was a response to having no ground truth. learning-harness Req 3.5, 8.2, 9.5 and the whole of the evolve tasks are retired, not amended.

2. Keep and port the three assets that stand on their own: the closed family taxonomy, the IR-driven candidate enumerator (96.6% ceiling), and the obligation / absence-bug checker. These become a Clearwing logic_auth + web_framework detection path that enumerates candidates deterministically and judges them once, rather than searching.

3. Ground truth for the web/logic classes that cannot crash a sandbox. This is the hard research problem and the real contribution. Options, to be decided in requirements:

   • a differential oracle — the vulnerable-parent vs fix-commit pair is the ground truth, exactly as our pair corpus intended, but the "reproduction" is a constructed request that the fixed version rejects and the vulnerable version serves (a sandboxed HTTP harness, the web analogue of poc_runner);
   • failing a runnable oracle, the fix-commit-as-blind-oracle discipline from Clearwing's FFmpeg protocol — run blind on the parent, score against the fix after the fact, never in context.

4. Corpus is harvested, not hand-labeled. Replace the pair-excerpt corpus with vulnerable-parent commits at repository scale (our existing recipes are a seed; the CVE-history and OSS-Fuzz harvesters are the model), and let the oracle — sandbox crash for memory, differential request for web — decide truth.

5. Reporting keeps the one honesty discipline we got right: per-slice numbers with the real-world slice deciding, the overfitting gap published, and the evidence level on every finding. Clearwing already emits an evidence ladder and SARIF; our per-slice + overfitting-gap discipline is additive.

6. Build vs. adopt — the honest recommendation

The instinct after a project like this is to rebuild. Resist it. This session logged eighteen defects, every one ours, three of them the same shape of fixing an instance and missing the class. A codebase we wrote from scratch to match Clearwing would carry that defect rate against a 19,000-line, MIT-licensed, battle-tested substrate that the maintainer already forks (norandom/deepwing).

Recommendation: do not rebuild. Adopt Clearwing (via the existing fork) as the execution substrate and contribute the web/logic detection path — taxonomy + IR candidate enumeration + obligation checker + the differential web oracle — as the piece it genuinely lacks. OpenUltraSAST's lasting output is then not a competing tool but the answer to Clearwing's open question: how do you ground-truth the dangerous web bugs that never crash?

The alternative — keep OpenUltraSAST standalone and bolt a sandbox onto it — means reimplementing Clearwing's sandbox lifecycle, container pooling, sanitizer images, PoC stability, campaign orchestration and disclosure workflow, which are the parts this session proved we get wrong. That is the expensive path and it competes where we are weakest.

Open questions for the requirements phase

• The differential web oracle is unproven. Can a sandboxed request harness reliably distinguish a vulnerable handler from its fixed twin for the access-control / injection / SSRF classes, at corpus scale, without per-repo setup? If not, the fix-commit-as-blind-oracle is the fallback, and it is weaker.
• Adoption mechanics. Contribute upstream to Lazarus-AI/clearwing, or maintain the web/logic path in the norandom/deepwing fork? The Upstream.md sync discipline already exists.
• What of the existing corpus survives as seed for the harvest, given the twin/straddle/tier defects the learning-harness reviews surfaced?
• Cost. Buttercup ran at ~$152/task; our web slice ran at $0.03–0.14 per correct pair. The differential oracle's sandbox cost is unmeasured and is the number that decides whether campaign-scale is affordable.

Sources: Big Sleep / SQLite zero-day · AIxCC finals — Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs · AIxCC results — 18 real 0-days, 43/54 patched · Clearwing · local: /home/mc/Source/clearwing, fork norandom/deepwing.