Open_Ultra_SAST/.kiro/specs/execution-grounded-detection/brief.md

# Corrective brief: execution-grounded detection

**Status:** course-setting. Supersedes `learning-harness` and `constrained-detector`. Not a requirements
document — a decision about what to build, what to delete, and what to adopt, written after the
learning-harness line produced numbers its own measurements say to discard.

## 1. The premise we got wrong

OpenUltraSAST was built on an unstated assumption: that an LLM is a *noisy oracle* whose judgement must be
averaged (K runs), floored (per-family noise budget), gated (sign test), and slowly improved (evolve loop).
Every mechanism in `learning-harness` follows from that assumption. This session measured all of it and it
does not hold:

- Five evolve rounds, $2.17, **zero decisions distinguishable from chance** — because the acceptance rule
  compares two ensemble votes over a detector that never runs the same investigation twice.
- 120 runs of one detector produced **120 distinct trajectories**. The variance is the loop's freedom to
  search, not the model's uncertainty about an answer.
- The noise floor a regression budget is derived from measures *how often an unconstrained search happens
  to reach the sink* — so the worse the search wanders, the more collateral damage a round is licensed to do.

The assumption is also empirically false as a claim about LLMs, and the evidence is now overwhelming:

- **Google Big Sleep** (Nov 2024): the first LLM-discovered zero-day in widely-used real-world software — a
  stack buffer underflow in SQLite — found by *variant analysis*, a sandboxed test case, and an
  auto-generated root-cause report. Not pattern-matching; reproduction.
- **DARPA AIxCC finals** (DEF CON 33, Aug 2025): seven autonomous systems processed 54M lines of code,
  patched 43 of 54 synthetic vulnerabilities and **uncovered 18 previously-unknown real 0-days**. Buttercup
  (Trail of Bits, 2nd place) found **28 vulnerabilities across 20 CWEs at 90% accuracy, $181 per point,
  using non-reasoning LLMs only**, ~$152 per task, 45 minutes each.

The lever in every one of those systems is **architecture, not prompt optimization**. Buttercup won with
non-reasoning models. Nobody evolved a prompt with a genetic loop. The question "can LLMs find security
bugs" was answered *yes* by 2024; we spent this project answering a question the field had closed, with the
wrong architecture.

## 2. What the successful systems share, and Clearwing is the open one

Big Sleep, Atlantis (AIxCC 1st), Buttercup (2nd), and **Clearwing** (`/home/mc/Source/clearwing`,
Lazarus-AI, MIT, an open reproduction of Anthropic's Glasswing) share one shape:

> **The LLM is a judgement layer on top of a deterministic + execution substrate. Truth comes from the
> sandbox — a crash, a sanitizer report, a patch that stops the crash — not from a score against a labeled
> corpus.**

Clearwing's evidence ladder is the whole thesis in one type:

```
suspicion → static_corroboration → crash_reproduced → root_cause_explained
          → exploit_demonstrated → patch_validated
```

The load-bearing gate is `poc_runner.py`: apply a candidate diff inside the sandbox, recompile, re-run the
PoC, and return a boolean for *does the crash still happen*. That single primitive replaces our entire
noise architecture. You do not average five noisy runs when a container can tell you whether the bug is
real. **We built K-runs and noise floors because we had no ground truth. They generate their own.**

## 3. What Clearwing does better, mapped to what failed here

| Our failure this session | What Clearwing does instead |
|---|---|
| K-runs / noise floor / sign test / regression budget — a statistical treatment for a structural problem | **Execution is the arbiter.** `poc_runner` reproduces or it doesn't. `stability.py` runs the PoC across N fresh containers (≥90% = stable) to kill flakiness at the source instead of averaging it. |
| Evolve loop: 5 rounds, 0 reliable decisions, $2.17 to learn nothing | **A fixed specialist roster** dispatched by file tags — `memory_safety`, `kernel_syscall`, `crypto_primitive`, `web_framework`, `logic_auth`, `general` — plus **variant analysis** (verify a bug, generate a grep+semantic pattern, find its structural twins, iterate to fixpoint). Density compounds *within a run*, no optimizer. |
| Second judge / fusion, gesturing at independence | **4-axis adversarial validator** (REAL / TRIGGERABLE / IMPACTFUL / GENERAL) that sees *only* the finding, PoC and exploit — never the discovery transcript — steel-mans both sides, and **must attempt reproduction**. |
| Corpus: 12 hand-harvested function excerpts per family, scored by text-inside-function, capped by twins/straddles/truncation defects we spent days repairing | **Corpus is real repos at vulnerable-parent commits + OSS-Fuzz** (7000 targets in `full` mode), run blind, with the fix commit used only as an *after-the-fact oracle*. Truth is generated by the sandbox, not annotated by us. `CampaignConfig` scales this to hundreds of repos with shared budget, checkpoint/resume, and diminishing-returns detection. |
| Detector chooses what to look at (555 grep calls vs 71 entry-point lookups) | **Deterministic preprocessing feeds candidates:** callgraph → intra-procedural taint (source→sink) → 3-axis ranker → hunter. The model judges; it does not search for the sink. This is exactly the `constrained-detector` design we had started — Clearwing is the mature version of it. |

## 4. The one thing OpenUltraSAST proved that Clearwing lacks

This is not a total loss, and the course is not "throw everything away." Two results here are real,
measured, and *absent from Clearwing*:

1. **IR-driven candidate enumeration with a published ceiling.** We measured that the pattern ruleset caps
   recall at 12.1% while call-sites-and-bindings from the semantic IR reach **96.6%** across the web slices,
   a median of ten sites per function. Clearwing's candidate feeder is C-first (taint tables for
   `memcpy`-shaped bugs); its `web_framework` specialist is a *single unconstrained prompt* with no
   equivalent candidate enumerator for logic/web classes.

2. **Absence-bug detection.** Our `obligations` work — an access-control bug is the *absence* of a guard
   around an operation its siblings protect — has no analogue in Clearwing, whose ladder is built around a
   *crash*. An IDOR or a missing authorization check never crashes a sandbox. Clearwing's execution
   ground-truth model does not reach these classes at all, and they are exactly the "dangerous but
   low-hanging" web bugs this project was chartered to catch.

So the gap Clearwing leaves open is **the web/logic taxonomy without a crash oracle** — which is where our
IR candidate enumerator, our family taxonomy, and our obligation checker are genuinely ahead.

## 5. The course

**Adopt Clearwing as the substrate; contribute the web/logic detection path it lacks; delete our noise
architecture entirely.**

Concretely, superseding both prior specs:

1. **Delete, do not fix.** The K-run machinery, the per-family noise floor, the regression budget, the sign
   test, the evolve loop, the acceptance rule, the mechanism-lever optimizer. All of it was a response to
   having no ground truth. `learning-harness` Req 3.5, 8.2, 9.5 and the whole of the evolve tasks are
   retired, not amended.

2. **Keep and port** the three assets that stand on their own: the closed family **taxonomy**, the
   IR-driven **candidate enumerator** (96.6% ceiling), and the **obligation / absence-bug checker**. These
   become a Clearwing `logic_auth` + `web_framework` detection path that enumerates candidates
   deterministically and judges them once, rather than searching.

3. **Ground truth for the web/logic classes that cannot crash a sandbox.** This is the hard research
   problem and the real contribution. Options, to be decided in requirements:
   - a *differential* oracle — the vulnerable-parent vs fix-commit pair is the ground truth, exactly as our
     pair corpus intended, but the "reproduction" is a constructed request that the fixed version rejects and
     the vulnerable version serves (a sandboxed HTTP harness, the web analogue of `poc_runner`);
   - failing a runnable oracle, the fix-commit-as-blind-oracle discipline from Clearwing's FFmpeg protocol —
     run blind on the parent, score against the fix *after the fact*, never in context.

4. **Corpus is harvested, not hand-labeled.** Replace the pair-excerpt corpus with vulnerable-parent
   commits at repository scale (our existing recipes are a seed; the CVE-history and OSS-Fuzz harvesters are
   the model), and let the oracle — sandbox crash for memory, differential request for web — decide truth.

5. **Reporting keeps the one honesty discipline we got right:** per-slice numbers with the real-world slice
   deciding, the overfitting gap published, and the evidence level on every finding. Clearwing already emits
   an evidence ladder and SARIF; our per-slice + overfitting-gap discipline is additive.

## 6. Build vs. adopt — the honest recommendation

The instinct after a project like this is to rebuild. Resist it. This session logged **eighteen defects,
every one ours**, three of them the same shape of fixing an instance and missing the class. A codebase we
wrote from scratch to match Clearwing would carry that defect rate against a 19,000-line, MIT-licensed,
battle-tested substrate that the maintainer *already forks* (`norandom/deepwing`).

**Recommendation:** do not rebuild. Adopt Clearwing (via the existing fork) as the execution substrate and
contribute the web/logic detection path — taxonomy + IR candidate enumeration + obligation checker + the
differential web oracle — as the piece it genuinely lacks. OpenUltraSAST's lasting output is then not a
competing tool but the answer to Clearwing's open question: *how do you ground-truth the dangerous web bugs
that never crash?*

The alternative — keep OpenUltraSAST standalone and bolt a sandbox onto it — means reimplementing
Clearwing's sandbox lifecycle, container pooling, sanitizer images, PoC stability, campaign orchestration
and disclosure workflow, which are the parts this session proved we get wrong. That is the expensive path
and it competes where we are weakest.

## Open questions for the requirements phase

- **The differential web oracle is unproven.** Can a sandboxed request harness reliably distinguish a
  vulnerable handler from its fixed twin for the access-control / injection / SSRF classes, at corpus scale,
  without per-repo setup? If not, the fix-commit-as-blind-oracle is the fallback, and it is weaker.
- **Adoption mechanics.** Contribute upstream to `Lazarus-AI/clearwing`, or maintain the web/logic path in
  the `norandom/deepwing` fork? The `Upstream.md` sync discipline already exists.
- **What of the existing corpus survives** as seed for the harvest, given the twin/straddle/tier defects the
  learning-harness reviews surfaced?
- **Cost.** Buttercup ran at ~$152/task; our web slice ran at $0.03–0.14 per correct pair. The differential
  oracle's sandbox cost is unmeasured and is the number that decides whether campaign-scale is affordable.

Sources: [Big Sleep / SQLite zero-day](https://thehackernews.com/2024/11/googles-ai-tool-big-sleep-finds-zero.html) ·
[AIxCC finals — Buttercup, 28 vulns / 20 CWEs / non-reasoning LLMs](https://blog.trailofbits.com/2025/08/09/trail-of-bits-buttercup-wins-2nd-place-in-aixcc-challenge/) ·
[AIxCC results — 18 real 0-days, 43/54 patched](https://semiengineering.com/aixcc-2025-what-it-means-for-device-security/) ·
[Clearwing](https://github.com/Lazarus-AI/clearwing) · local: `/home/mc/Source/clearwing`, fork `norandom/deepwing`.