marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-18 20:49:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(learning-harness): fold four research threads into the brief and add research.md

Browse Source 
AIxCC cyber reasoning systems, self-improving harness methods (GEPA, DGM,
Self-Harness, SkillOpt, AHE), vulnerability-dataset pitfalls and CWE taxonomies,
and routing/regression-control literature. The design changes they force:
coarse verifier-aligned families instead of fine CWE, multi-label routing with
an unknown class, pair-wise outcomes with a directional-bias index, K-run
reliable-change tests, a per-class seed-noise floor as the regression budget,
a no-regression acceptance rule with ties rejected, structural edits over prose
edits, a proposer blind to the holdout, and verifier code and gold labels
outside every editable surface.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-06 08:22:43 +02:00
co-authored by Claude Fable 5.1
parent 3259d0469f
commit ea05e3357c
2 changed files with 104 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/learning-harness/research.md
+51
View File
@@ -0,0 +1,51 @@
# Research: learning-harness
Web research conducted 2026-09-06 in four threads by research subagents; summaries are theirs, condensed, with primary sources. Speculation is marked. This file backs the "What others learned" section of `brief.md`.
## 1. Cyber reasoning systems (AIxCC finals 2025, Big Sleep, XBOW, Glasswing)
Primary cross-team source: USENIX Security '26 SoK on AIxCC, arXiv 2602.07666. 42-b3yond-6ug and Lacrosse have no first-party write-up.
| System | Decomposition / routing key | LLM vs tools |
|---|---|---|
| Atlantis (1st), arXiv 2509.14589, team-atlanta.github.io/blog/post-afc | N-version sub-CRSs by language (C, Java, Multilang) with orthogonal designs; six patch agents in parallel, no routing ("no single agent consistently outperformed"); rate limit per fuzzer×sanitizer; Java sinks keyed by Jazzer sanitizer class | Three tiers: LLM-augmented (seeds), LLM-opinionated (hints as optimizations, never correctness), LLM-driven (multi-agent call-graph, bug candidates, blobs). Fuzzers/concolic are ground truth |
| Buttercup (2nd), blog.trailofbits.com 2025-08-08/09 | Seven patch agents by role (RCA, retrieval, writer, reflection); language tooling and CWE guidance in prompts | Fuzzers discover; tree-sitter/CodeQuery program model; cheap non-reasoning LLMs only ($21k for 100k calls); >90% accuracy |
| RoboDuck (3rd), theori.io/blog/aixcc-and-roboduck-63447, /building-effective-llm-agents-63446 | Bug candidate is the anchor: Infer + LLM static analysis + crashes → logprob-scored binary classifier (~$0.001/report) → top 20% to a source-browsing agent (~$0.50) | LLM-first PoVs without fuzzing; three models race, first verified wins; Infer ~99.9% FP |
| FuzzingBrain (4th), arXiv 2509.07225 | 23 strategies keyed on mode × language × stage; per-language CWE lists aligned to sanitizers | LLM-dominant; libFuzzer only; nothing reported unless a sanitizer fires on a real build |
| ARTIPHISHELL (5th), support.shellphish.net 2025-08-22 postmortem | 53 components; multi-source voting; dual patchers | PoV submitted only after five consecutive local reproductions |
| Big Sleep / Naptime, projectzero.google 2024/06 and 2024/10 | No classifier; variant analysis seeded from recent commits | Single agent with code browser, Python, debugger, reporter; must reproduce a crash; parallel independent trajectories |
| XBOW, xbow.com/blog/top-1-how-xbow-did-it | Coordinator spawns per-vulnerability-class solvers; validators keyed by class (canary, headless browser for XSS, timing for blind SQLi) | The non-AI validator "is the only one who can decide if the agent succeeded" |
| Anthropic Glasswing, anthropic.com/research/glasswing-initial-update | Threat-model builder prioritizes targets; scanning subagents; reusable skills | 90.6% of 1,752 findings valid after human reproduction; internals unpublished |
Lessons: (1) route on mode × language × sanitizer/CWE family, never fine CWE; route to what the verifier can check. (2) Only proof survives; the accuracy multiplier (Atlantis 0.9999 vs 0.9044) decided the ranking; RoboDuck lost 16.3 points to FPs. (3) Cheap classifier → expensive agent → verifier (RoboDuck 500× cost tiering; logprob averaging for stable scores). (4) Ensembles beat selection when the winner is unpredictable; attribution per agent done by hand post hoc. (5) Dedup granularity is a scoring bug (Shadowsocks: five same-pattern bugs differing by line). (6) Validated ≠ correct: 38–46% of automatically accepted patches were semantically wrong (SoK KF4). (7) Hallucination controls are structural: forced tool calls, curated tools instead of bash, symbol resolution via Joern/LSP. (8) Stability beat cleverness: budget caps, cached container state, exit-code-0 logging, lost logs sank teams. (9) Cross-run learning nearly absent; Atlantis's only trained component (GRPO retrieval, C only) showed catastrophic forgetting across projects; daily leakage-resistant CI benchmarks instead. (10) Cost is a control: >$1,000/hour unbounded (arXiv 2603.08566); Buttercup $181/point.
## 2. Self-improving harness methods
| Method | Evolved unit | Search operator | Validation |
|---|---|---|---|
| DSPy MIPROv2 (dspy.ai) | Few-shot demos + instructions per module | Bootstrapped traces; LLM proposals; Bayesian optimization | Minibatch trials, full validation every 5 steps; ≥200 examples advised |
| GEPA, arXiv 2507.19457 | One module's prompt per iteration | Natural-language reflection on trajectories; system-aware merge | Minibatch gate → full per-instance Pareto front; candidates sampled ∝ instances led |
| TextGrad; PrefPO critique arXiv 2603.19311 | Prompt text | Textual gradients | Validation revert; 86% of TextGrad prompts hacked/brittle, 14.7× length inflation |
| OPRO 2309.03409 / EvoPrompt 2309.08532 / PromptBreeder 2309.16797 | Task prompt | Proposal / GA / self-referential mutation | Train-score selection; EvoPrompt degrades to generic prose on structured tasks |
| AlphaEvolve (DeepMind blog 2025) | Code diffs | LLM mutation over MAP-Elites program DB | Evaluator cascade; needs automatically verifiable problems |
| Darwin Gödel Machine, arXiv 2505.22954, sakana.ai/dgm | Agent code/tools | Archive sampling ∝ score, ∝ 1/(1+children) | Staged 10→60→200 tasks; held-out models/languages; faked test logs when checker visible |
| ADAS, arXiv 2408.08435 | Agent code | Meta-agent over archive | Held-out validation and domains |
| Reflexion / ExpeL 2308.10144 / AWM 2409.07429 / Voyager 2305.16291 | Reflections / insights / workflows / skills | Self-critique; contrast; induction | Mostly none |
| Self-Harness, arXiv 2606.09498 | Harness components | Weakness mining → K minimal proposals | Accept iff Δ_in ≥ 0 ∧ Δ_ho ≥ 0 ∧ one > 0 |
| SkillOpt, arXiv 2605.23904 | SKILL.md | Reflection with rejected-edit buffer | Strict improvement on held-out; ties rejected |
| AHE, arXiv 2604.25850 | Tools, middleware, memory, prompts | Evidence-driven edits with per-edit manifest (predicted fixes and at-risk regressions) | Prediction vs observed deltas; rollback |
| Meta-Harness, arXiv 2603.28052 | Harness code | Agentic proposer over prior candidates and traces | Held-out models and tasks |
Lessons: (1) per-instance/per-class tracking doubled gains vs greedy best-overall (GEPA +12.44% vs +6.05%). (2) No-regression acceptance rule; reject ties (Self-Harness, SkillOpt). (3) Regression blindness: AHE fix-prediction precision 33.7% but regression prediction 11.8%; the cross-class sweep is mandatory. (4) Two-stage evaluation is where sample efficiency comes from; most budget is validation. (5) Prompt bloat is the overfit signature (Decagon: +75% length, −2% accuracy; cap ~1,500 chars gave 4× compression at −0.8%). (6) Keep the reflector blind to validation; keep a rejected-edit buffer (arXiv 2602.22483; SkillOpt). (7) Structural edits transfer (+5.1–10.1 points across model families), system-prompt-only edits regressed −2.3 (AHE ablation). (8) Self-diagnosis confabulates: 0/121 Reflexion reflections correct; structured trajectory facts → 86% (arXiv 2605.29463). (9) Hide the checker (DGM). (10) One objective per round (arXiv 2605.26046: joint criteria −59% task focus). (11) Audit that a skill beats re-showing the failure (ContinualSkillBench arXiv 2608.03874). (12) Harness-edit quality flat across model tiers; spend on the detector (arXiv 2605.30621). Speculation: no source measures class-to-class oscillation; per-class Pareto retention plus the acceptance rule should bound it. Anthropic's Claude Code postmortem (InfoQ 2026-05: three unattributed harness changes → six-week degradation) is the cautionary tale.
## 3. Dataset pitfalls and class taxonomies
Scoring: (1) dedup by normalized body hash before splitting; split by commit, chronologically (PrimeVul arXiv 2403.18624; test-set copies 12.7–18.9% in older sets). (2) Automatic labels 25–60% correct (BigVul 25%, CVEfixes 51.7%, DiverseVul 60%; Croft et al. ICSE'23 20–71% wrong, 17–99% duplicates; CleanVul arXiv 2411.17274). (3) Pair-wise outcomes P-C/P-V/P-B/P-R, not F1 (GPT-4 CoT P-C 12.94% vs P-V 54.26%; JitVul ACL 2025: agents lose 8–9 points of pair accuracy while "winning" F1; Vul-RAG pairwise 0.21 vs plain 0.61). (4) Directional-bias index (CWE-Trace arXiv 2606.20502: −85.5 to +94.8 points at ~50% accuracy). (5) Score at fixed FPR (PrimeVul VD-S; VFC benchmark arXiv 2605.13138). (6) Match on location and class with hierarchical credit and hallucination penalty (CASTLE arXiv 2503.09433; ALPHA arXiv 2601.01320). (7) Robustness to renames/whitespace/dead code (SecLLMHolmes arXiv 2312.12575; patched-code FPs dominate). (8) CVE-date contamination checks are weak evidence, keep a post-cutoff slice anyway (thin: one paper). (9) Function-level pairs lose context: up to 54% of patched-code misclassification (arXiv 2504.13474). (10) Run-to-run consistency 8.3–81.9% by model (ALPHA).
Taxonomy: (1) fine CWE from code unsolved (top-1 ≤ 14.7% over 74 CWEs; 86.5% lateral errors; VulDetectBench arXiv 2406.07595 >80% coarse, <30% detailed). (2) From CVE text ~85–90% ceiling (arXiv 2603.14911; CweAgent arXiv 2608.21977). (3) Human/NVD disagreement 15–50%; CWE-699 families as "defensible" (Seclometry; Sonnet vs NVD 73.1% exact, 84.5% with parent–child, LLM right in 72/100 disagreements). (4) Score hierarchically. (5) Few aggregated classes: MoEVD arXiv 2501.16454 collapsed 88 → 12 families, per-class experts +12.8% F1, router 63.8% right, F1 0.18 when wrong; three-class OOB/UAF/other 87.4% (arXiv 2509.22796). (6) Specialists need hard negatives (arXiv 2408.02329: F1 0.89 in class → 0.12–0.19 on full corpus). (7) Per-class detectors work when the class fixes sources/sinks (IRIS arXiv 2405.17238: 55 vs 27 CodeQL detections, FDR 46–85%). (8) VFC typing immature (F1 0.53–0.70). Speculation: 8–12 hierarchy-derived families with partial credit beats fine CWEs for this corpus.
## 4. Routing and regression control
Router: (1) learned routers plateau 10–20 points below oracle, sometimes lose to random (RouteLLM arXiv 2406.18665; RouterBench arXiv 2403.12031; arXiv 2606.07587). (2) Evaluate router AUROC separately from expert strength (arXiv 2602.11877). (3) Small embedding classifier fine-tuned on own labels: 80.4% → 98.5% (Red Hat vLLM Semantic Router, 2026-06). (4) Cascades tolerate router error until judge error > 0.2 (FrugalGPT arXiv 2305.05176; RouterBench). (5) Multi-expert routing with balance (Expert-Choice, Google; AgentRouter arXiv 2510.05445); speculation: multi-label with `unknown` is safer than argmax for SAST.
Attribution: (6) slice everything (Oakden-Rayner arXiv 1909.12475; Slice-based learning 1909.06349; Overton 1909.05372; Model Cards 1810.03993). (7) Negative flips per item (PCT arXiv 2011.09161: 5.2% NFR at equal accuracy; MUSCLE 2407.09435: 8–49% NFR across Llama versions; arXiv 2604.27405: +1.6–2.8 point gains masked 28–39% items worse, single-shot eval missed 42% of changes → K ≥ 3 samples and a reliable-change test; UpgradeBench 2608.20918: neutral retraining breaks 1–5%). (8) Frozen per-class configurations (+20%, Progressive Prompts arXiv 2301.12314); self-feedback-only skill loops drift, external feedback does not (SkillLearnBench 2604.20087); self-revision flips correct → incorrect more often than the reverse (arXiv 2310.01798). (9) Per-instance archives over one champion (GEPA; DGM lineage logs caught faked tests). (10) Conjunctive per-slice gates with relative thresholds vs the last blessed baseline (TFMA model validations; ease.ml/ci arXiv 1903.00278; Amazon Alexa regression-free updates). No public numeric NFR budget; speculation: the measured seed-noise floor per class is the defensible budget.