Open_Ultra_SAST/.kiro/specs/learning-harness/research.md

Research: learning-harness

Web research conducted 2026-09-06 in four threads by research subagents; summaries are theirs, condensed, with primary sources. Speculation is marked. This file backs the "What others learned" section of brief.md.

1. Cyber reasoning systems (AIxCC finals 2025, Big Sleep, XBOW, Glasswing)

Primary cross-team source: USENIX Security '26 SoK on AIxCC, arXiv 2602.07666. 42-b3yond-6ug and Lacrosse have no first-party write-up.

System	Decomposition / routing key	LLM vs tools
Atlantis (1st), arXiv 2509.14589, team-atlanta.github.io/blog/post-afc	N-version sub-CRSs by language (C, Java, Multilang) with orthogonal designs; six patch agents in parallel, no routing ("no single agent consistently outperformed"); rate limit per fuzzer×sanitizer; Java sinks keyed by Jazzer sanitizer class	Three tiers: LLM-augmented (seeds), LLM-opinionated (hints as optimizations, never correctness), LLM-driven (multi-agent call-graph, bug candidates, blobs). Fuzzers/concolic are ground truth
Buttercup (2nd), blog.trailofbits.com 2025-08-08/09	Seven patch agents by role (RCA, retrieval, writer, reflection); language tooling and CWE guidance in prompts	Fuzzers discover; tree-sitter/CodeQuery program model; cheap non-reasoning LLMs only ($21k for 100k calls); >90% accuracy
RoboDuck (3rd), theori.io/blog/aixcc-and-roboduck-63447, /building-effective-llm-agents-63446	Bug candidate is the anchor: Infer + LLM static analysis + crashes → logprob-scored binary classifier ($0.001/report) → top 20% to a source-browsing agent ($0.50)	LLM-first PoVs without fuzzing; three models race, first verified wins; Infer ~99.9% FP
FuzzingBrain (4th), arXiv 2509.07225	23 strategies keyed on mode × language × stage; per-language CWE lists aligned to sanitizers	LLM-dominant; libFuzzer only; nothing reported unless a sanitizer fires on a real build
ARTIPHISHELL (5th), support.shellphish.net 2025-08-22 postmortem	53 components; multi-source voting; dual patchers	PoV submitted only after five consecutive local reproductions
Big Sleep / Naptime, projectzero.google 2024/06 and 2024/10	No classifier; variant analysis seeded from recent commits	Single agent with code browser, Python, debugger, reporter; must reproduce a crash; parallel independent trajectories
XBOW, xbow.com/blog/top-1-how-xbow-did-it	Coordinator spawns per-vulnerability-class solvers; validators keyed by class (canary, headless browser for XSS, timing for blind SQLi)	The non-AI validator "is the only one who can decide if the agent succeeded"
Anthropic Glasswing, anthropic.com/research/glasswing-initial-update	Threat-model builder prioritizes targets; scanning subagents; reusable skills	90.6% of 1,752 findings valid after human reproduction; internals unpublished

Lessons: (1) route on mode × language × sanitizer/CWE family, never fine CWE; route to what the verifier can check. (2) Only proof survives; the accuracy multiplier (Atlantis 0.9999 vs 0.9044) decided the ranking; RoboDuck lost 16.3 points to FPs. (3) Cheap classifier → expensive agent → verifier (RoboDuck 500× cost tiering; logprob averaging for stable scores). (4) Ensembles beat selection when the winner is unpredictable; attribution per agent done by hand post hoc. (5) Dedup granularity is a scoring bug (Shadowsocks: five same-pattern bugs differing by line). (6) Validated ≠ correct: 38–46% of automatically accepted patches were semantically wrong (SoK KF4). (7) Hallucination controls are structural: forced tool calls, curated tools instead of bash, symbol resolution via Joern/LSP. (8) Stability beat cleverness: budget caps, cached container state, exit-code-0 logging, lost logs sank teams. (9) Cross-run learning nearly absent; Atlantis's only trained component (GRPO retrieval, C only) showed catastrophic forgetting across projects; daily leakage-resistant CI benchmarks instead. (10) Cost is a control: >$1,000/hour unbounded (arXiv 2603.08566); Buttercup $181/point.

2. Self-improving harness methods

Method	Evolved unit	Search operator	Validation
DSPy MIPROv2 (dspy.ai)	Few-shot demos + instructions per module	Bootstrapped traces; LLM proposals; Bayesian optimization	Minibatch trials, full validation every 5 steps; ≥200 examples advised
GEPA, arXiv 2507.19457	One module's prompt per iteration	Natural-language reflection on trajectories; system-aware merge	Minibatch gate → full per-instance Pareto front; candidates sampled ∝ instances led
TextGrad; PrefPO critique arXiv 2603.19311	Prompt text	Textual gradients	Validation revert; 86% of TextGrad prompts hacked/brittle, 14.7× length inflation
OPRO 2309.03409 / EvoPrompt 2309.08532 / PromptBreeder 2309.16797	Task prompt	Proposal / GA / self-referential mutation	Train-score selection; EvoPrompt degrades to generic prose on structured tasks
AlphaEvolve (DeepMind blog 2025)	Code diffs	LLM mutation over MAP-Elites program DB	Evaluator cascade; needs automatically verifiable problems
Darwin Gödel Machine, arXiv 2505.22954, sakana.ai/dgm	Agent code/tools	Archive sampling ∝ score, ∝ 1/(1+children)	Staged 10→60→200 tasks; held-out models/languages; faked test logs when checker visible
ADAS, arXiv 2408.08435	Agent code	Meta-agent over archive	Held-out validation and domains
Reflexion / ExpeL 2308.10144 / AWM 2409.07429 / Voyager 2305.16291	Reflections / insights / workflows / skills	Self-critique; contrast; induction	Mostly none
Self-Harness, arXiv 2606.09498	Harness components	Weakness mining → K minimal proposals	Accept iff Δ_in ≥ 0 ∧ Δ_ho ≥ 0 ∧ one > 0
SkillOpt, arXiv 2605.23904	SKILL.md	Reflection with rejected-edit buffer	Strict improvement on held-out; ties rejected
AHE, arXiv 2604.25850	Tools, middleware, memory, prompts	Evidence-driven edits with per-edit manifest (predicted fixes and at-risk regressions)	Prediction vs observed deltas; rollback
Meta-Harness, arXiv 2603.28052	Harness code	Agentic proposer over prior candidates and traces	Held-out models and tasks

Lessons: (1) per-instance/per-class tracking doubled gains vs greedy best-overall (GEPA +12.44% vs +6.05%). (2) No-regression acceptance rule; reject ties (Self-Harness, SkillOpt). (3) Regression blindness: AHE fix-prediction precision 33.7% but regression prediction 11.8%; the cross-class sweep is mandatory. (4) Two-stage evaluation is where sample efficiency comes from; most budget is validation. (5) Prompt bloat is the overfit signature (Decagon: +75% length, −2% accuracy; cap ~1,500 chars gave 4× compression at −0.8%). (6) Keep the reflector blind to validation; keep a rejected-edit buffer (arXiv 2602.22483; SkillOpt). (7) Structural edits transfer (+5.1–10.1 points across model families), system-prompt-only edits regressed −2.3 (AHE ablation). (8) Self-diagnosis confabulates: 0/121 Reflexion reflections correct; structured trajectory facts → 86% (arXiv 2605.29463). (9) Hide the checker (DGM). (10) One objective per round (arXiv 2605.26046: joint criteria −59% task focus). (11) Audit that a skill beats re-showing the failure (ContinualSkillBench arXiv 2608.03874). (12) Harness-edit quality flat across model tiers; spend on the detector (arXiv 2605.30621). Speculation: no source measures class-to-class oscillation; per-class Pareto retention plus the acceptance rule should bound it. Anthropic's Claude Code postmortem (InfoQ 2026-05: three unattributed harness changes → six-week degradation) is the cautionary tale.

3. Dataset pitfalls and class taxonomies

Scoring: (1) dedup by normalized body hash before splitting; split by commit, chronologically (PrimeVul arXiv 2403.18624; test-set copies 12.7–18.9% in older sets). (2) Automatic labels 25–60% correct (BigVul 25%, CVEfixes 51.7%, DiverseVul 60%; Croft et al. ICSE'23 20–71% wrong, 17–99% duplicates; CleanVul arXiv 2411.17274). (3) Pair-wise outcomes P-C/P-V/P-B/P-R, not F1 (GPT-4 CoT P-C 12.94% vs P-V 54.26%; JitVul ACL 2025: agents lose 8–9 points of pair accuracy while "winning" F1; Vul-RAG pairwise 0.21 vs plain 0.61). (4) Directional-bias index (CWE-Trace arXiv 2606.20502: −85.5 to +94.8 points at ~50% accuracy). (5) Score at fixed FPR (PrimeVul VD-S; VFC benchmark arXiv 2605.13138). (6) Match on location and class with hierarchical credit and hallucination penalty (CASTLE arXiv 2503.09433; ALPHA arXiv 2601.01320). (7) Robustness to renames/whitespace/dead code (SecLLMHolmes arXiv 2312.12575; patched-code FPs dominate). (8) CVE-date contamination checks are weak evidence, keep a post-cutoff slice anyway (thin: one paper). (9) Function-level pairs lose context: up to 54% of patched-code misclassification (arXiv 2504.13474). (10) Run-to-run consistency 8.3–81.9% by model (ALPHA).

Taxonomy: (1) fine CWE from code unsolved (top-1 ≤ 14.7% over 74 CWEs; 86.5% lateral errors; VulDetectBench arXiv 2406.07595 >80% coarse, <30% detailed). (2) From CVE text ~85–90% ceiling (arXiv 2603.14911; CweAgent arXiv 2608.21977). (3) Human/NVD disagreement 15–50%; CWE-699 families as "defensible" (Seclometry; Sonnet vs NVD 73.1% exact, 84.5% with parent–child, LLM right in 72/100 disagreements). (4) Score hierarchically. (5) Few aggregated classes: MoEVD arXiv 2501.16454 collapsed 88 → 12 families, per-class experts +12.8% F1, router 63.8% right, F1 0.18 when wrong; three-class OOB/UAF/other 87.4% (arXiv 2509.22796). (6) Specialists need hard negatives (arXiv 2408.02329: F1 0.89 in class → 0.12–0.19 on full corpus). (7) Per-class detectors work when the class fixes sources/sinks (IRIS arXiv 2405.17238: 55 vs 27 CodeQL detections, FDR 46–85%). (8) VFC typing immature (F1 0.53–0.70). Speculation: 8–12 hierarchy-derived families with partial credit beats fine CWEs for this corpus.

4. Routing and regression control

Router: (1) learned routers plateau 10–20 points below oracle, sometimes lose to random (RouteLLM arXiv 2406.18665; RouterBench arXiv 2403.12031; arXiv 2606.07587). (2) Evaluate router AUROC separately from expert strength (arXiv 2602.11877). (3) Small embedding classifier fine-tuned on own labels: 80.4% → 98.5% (Red Hat vLLM Semantic Router, 2026-06). (4) Cascades tolerate router error until judge error > 0.2 (FrugalGPT arXiv 2305.05176; RouterBench). (5) Multi-expert routing with balance (Expert-Choice, Google; AgentRouter arXiv 2510.05445); speculation: multi-label with unknown is safer than argmax for SAST.

Attribution: (6) slice everything (Oakden-Rayner arXiv 1909.12475; Slice-based learning 1909.06349; Overton 1909.05372; Model Cards 1810.03993). (7) Negative flips per item (PCT arXiv 2011.09161: 5.2% NFR at equal accuracy; MUSCLE 2407.09435: 8–49% NFR across Llama versions; arXiv 2604.27405: +1.6–2.8 point gains masked 28–39% items worse, single-shot eval missed 42% of changes → K ≥ 3 samples and a reliable-change test; UpgradeBench 2608.20918: neutral retraining breaks 1–5%). (8) Frozen per-class configurations (+20%, Progressive Prompts arXiv 2301.12314); self-feedback-only skill loops drift, external feedback does not (SkillLearnBench 2604.20087); self-revision flips correct → incorrect more often than the reverse (arXiv 2310.01798). (9) Per-instance archives over one champion (GEPA; DGM lineage logs caught faked tests). (10) Conjunctive per-slice gates with relative thresholds vs the last blessed baseline (TFMA model validations; ease.ml/ci arXiv 1903.00278; Amazon Alexa regression-free updates). No public numeric NFR budget; speculation: the measured seed-noise floor per class is the defensible budget.