From ea05e3357c19ee153a1b1d66991519255b70103c Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Sun, 6 Sep 2026 08:22:43 +0200 Subject: [PATCH] docs(learning-harness): fold four research threads into the brief and add research.md AIxCC cyber reasoning systems, self-improving harness methods (GEPA, DGM, Self-Harness, SkillOpt, AHE), vulnerability-dataset pitfalls and CWE taxonomies, and routing/regression-control literature. The design changes they force: coarse verifier-aligned families instead of fine CWE, multi-label routing with an unknown class, pair-wise outcomes with a directional-bias index, K-run reliable-change tests, a per-class seed-noise floor as the regression budget, a no-regression acceptance rule with ties rejected, structural edits over prose edits, a proposer blind to the holdout, and verifier code and gold labels outside every editable surface. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG --- .kiro/specs/learning-harness/brief.md | 91 ++++++++++++++---------- .kiro/specs/learning-harness/research.md | 51 +++++++++++++ 2 files changed, 104 insertions(+), 38 deletions(-) create mode 100644 .kiro/specs/learning-harness/research.md diff --git a/.kiro/specs/learning-harness/brief.md b/.kiro/specs/learning-harness/brief.md index a39a229..fd870d1 100644 --- a/.kiro/specs/learning-harness/brief.md +++ b/.kiro/specs/learning-harness/brief.md @@ -11,80 +11,95 @@ Measured 2026-09-06 on the vibe-py holdout (17 vendored pairs), the existing too | tool hunter | 0/17 correct, Youden −0.588 | **12/14 correct, Youden +0.857, 0 leaks** | | overlay (facts + shapes) | 5/17 correct, Youden +0.294 | 5/14 correct, Youden +0.357 | -The published hunter number is a scoring artifact: the pair scorer counts a hunter finding only when the finding text contains the literal CWE id and the prompt never asks for one, so 24 of 25 vulnerable-side findings that sit inside the labeled function and name the labeled bug (SQL injection in `resolve_pastes`, BOLA in `get_by_title`, SSTI in `read_root`, missing authorization on the profile lookup) were thrown away. The 24 "leaks" split into 3 findings on byte-identical twins (unscorable by construction) and 21 findings outside the labeled function on the trap twin, most of them real defects in a deliberately vulnerable repository. The two remaining misses are `before_request` (`loads(b64decode(cookie))` with no import in the excerpt) and `bad_mvc.do_GET` (labeled CWE-79, the hunter reported the SQL concatenation on the same lines; the label may be wrong). +The published hunter number is a scoring artifact: the pair scorer counts a hunter finding only when the finding text contains the literal CWE id and the prompt never asks for one, so 24 of 25 vulnerable-side findings that sit inside the labeled function and name the labeled bug were thrown away. The 24 "leaks" split into 3 findings on byte-identical twins and 21 findings outside the labeled function on the trap twin, most of them real defects in a deliberately vulnerable repository. Raw runs: `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-*.json`. Three consequences follow. -1. **The model with tools is already the best detector we own, by a wide margin, without any learning.** The roadmap thesis ("recall from the model with tools plus proof; precision and silence from static checks") is confirmed, and the harness has been optimizing the wrong component. -2. **Detection, silence and leak are meaningless without a class.** The same finding is a detection in one class, a leak in another, and noise in a third. Every scorer, lever and gate in the tool currently uses text tokens (CWE strings, sink names) as a proxy for class, and the corpus's own class labels are 32 free-text strings across 7 slices, with 130 of 176 vfc rows labeled `other`. -3. **Improvements without a class dimension oscillate.** A change that helps SQL injection and hurts access control reads as "no change" on an aggregate Youden. The loop then reverts it, or keeps it, for the wrong reason, and the system degrades in ways nobody predicted. This is the maintainer's observation and it is exactly what the corpus-seeded lever does today. +1. **The model with tools is already the best detector we own, without any learning.** The roadmap thesis ("recall from the model with tools plus proof; precision and silence from static checks") is confirmed, and the harness has been optimizing the wrong component. +2. **Detection, silence and leak are meaningless without a class.** Every scorer, lever and gate uses text tokens (CWE strings, sink names) as a proxy for class; the corpus's own labels are 32 free-text strings across 7 slices, and 130 of 176 vfc rows say `other`. +3. **Improvements without a class dimension oscillate.** A change that helps one class and hurts another reads as no change on an aggregate Youden; the loop keeps or reverts it for the wrong reason and the system degrades in ways nobody predicted. This is the maintainer's observation and it is what the corpus-seeded lever does today. -Repairs already measured and required regardless (from the folded `closed-loop-integrity` brief): the mechanism lever seeds candidates from holdout pairs (holdout Youden +0.059 with every pair teaching, −0.059 with the train split only); five vibe-py twins are byte-identical; TypeScript has no grammar; absence excerpts carry no registration context; unit fixtures are easier than the corpus rows they stand in for. +Repairs already measured and required regardless: the mechanism lever seeds candidates from holdout pairs (holdout Youden +0.059 with every pair teaching, −0.059 with the train split only); five vibe-py twins are byte-identical; TypeScript has no grammar; absence excerpts carry no registration context; unit fixtures are easier than the corpus rows they stand in for. + +## What others learned + +Full sourced notes in `research.md` (four threads, 2026-09-06). The findings that change this design: + +**Cyber reasoning systems (AIxCC finals, Big Sleep, XBOW, Glasswing).** The routing key that worked in every finalist was mode × language × sanitizer or CWE family, aligned to what the oracle can confirm; nobody routed on fine CWE. Only proof survives: every top system gates a report on an executed oracle, the accuracy multiplier decided the ranking (Atlantis 0.9999 versus 0.9044), and XBOW keys a non-AI validator per vulnerability class that alone decides success. RoboDuck's cheap logprob classifier (~$0.001) sending the top 20% to a $0.50 agent is the closest public analogue of an auto-classifier front end. Ensembles beat selection when the winner is unpredictable, with attribution done per agent after the fact. Validated is not correct: 38–46% of automatically accepted patches were semantically wrong. Hallucination controls are structural (forced tool calls, curated tools instead of bash, symbol resolution). Cross-run learning is nearly absent in the field, and the one trained component reported catastrophic forgetting across projects. Stability and cost decided more than cleverness. + +**Self-improving harnesses (GEPA, DSPy, AlphaEvolve, DGM, Self-Harness, SkillOpt, AHE, Meta-Harness).** Track wins per instance or class, never the aggregate: GEPA's per-instance Pareto archive doubled the gain of greedy selection. Accept a change only when train and holdout are both non-negative and one is strictly positive; reject ties. Meta-agents predict the class they help at 33.7% precision and the class they hurt at 11.8% ("regression blindness"), so a cross-class sweep cannot be optional. Two-stage evaluation is where sample efficiency comes from. Prompt bloat is the overfit signature; cap length. Structural edits (tools, checklists, memory) transfer across model families, prose edits regress. Self-diagnosis confabulates (0 of 121 reflections named the real cause; structured trajectory facts reached 86%). Hide the checker and the gold labels from the editable surface. One change, one class per round. Harness-edit quality is flat across model tiers while detector quality is not: a cheap meta-agent, spend on the detector. + +**Datasets and taxonomies.** Deduplicate by normalized hash before splitting and split by repository, chronologically. Report pair-wise outcomes (pair-correct, both-vulnerable, both-benign, reversed) and a directional-bias index, not F1; models that flag both sides win F1 while losing pair accuracy. Match on location and class with hierarchical partial credit and the maximum penalty for a hallucinated class. Fine CWE classification from code is unsolved (top-1 ≤ 14.7%, most errors lateral siblings); coarse families are workable (88 CWEs collapsed to 12 gave per-class experts +12.8% F1, with a router right 63.8% of the time and F1 0.18 when wrong). Specialists collapse without hard negatives. Verdicts flip on renames and whitespace; single-shot evaluation missed 42% of real changes, so evaluate with K ≥ 3 runs and a reliable-change test. Function-level pairs lose context, the same defect as our context-free absence excerpts. + +**Routing and regression control.** Learned routers plateau 10–20 points below an oracle and can lose to random; measure router accuracy on its own against random and oracle. Route to several experts with an explicit `unknown`, never argmax. Slice everything; aggregate gains of +2 points have masked 28–39% of items getting worse. Freeze per-class configurations to prevent interference; skill loops driven by self-feedback alone drift, external feedback does not. Production gates are conjunctive per slice with relative thresholds against the last blessed baseline; no public numeric negative-flip budget exists, so the defensible budget is the measured seed-noise floor per class. ## Desired Outcome -A harness that improves a **set of classified detectors** round by round, where every change is attributed to the class it was meant to move, every unpredicted regression in another class counts against it, and nothing is published without its unscorable denominator. +A harness that improves a **set of classified detectors** round by round, where every change is attributed to the class it was meant to move, every unpredicted regression in another class counts against it, only what a verifier can confirm is reported above suspicion, and nothing is published without its unscorable denominator and its noise floor. -- **Auto-classifier.** Every unit of work (a labeled pair, a finding, a scan target region) is assigned to a class from a closed, versioned taxonomy by a classifier that is itself measured: agreement with maintainer labels on the reviewed pairs, confusion matrix per round. The classifier is the routing key for everything downstream: which detector runs, which skills load, which holdout the change is validated on, which journal row records it. -- **Per-class detectors.** One hunter configuration per class: system prompt, skills, tool set, step budget, evidence checklist, known counterexamples. A round evolves one class's configuration and is validated on that class's holdout plus a regression sweep over every other class. The static machinery (facts, overlay, obligations, dominance, sandbox) becomes tools the detector calls and verifiers that check its claims; it stops being the detector. -- **Class-aware scorer.** Detection = a finding in the labeled function whose class matches the label. Silence = no finding of the labeled class in the labeled function on the fixed side. Unscorable rows (identical twin, unsupported language, unresolved label) are listed by reason and removed from the denominator, never deleted. The text-token proxy goes away. -- **Learning unit = skill, not regex.** A round's output is a natural-language skill, a tool recipe, a checklist or a counterexample, drafted from a failure analysis, tested on the class's train split, validated on its holdout, journaled with a hypothesis, the predicted affected class, and the measured attribution. HarnessX's `MetaAgent.evolve` with its journal (`hypothesis_id`, `levers`, `predicted_affected`, precision discounted by `regressed_unpredicted`) is the loop primitive; the tool supplies tasks, trajectories, the evaluator and the class routing. -- **Honest fitness.** Split enforced by construction (a candidate touching a holdout pair is refused, not scored); per-class Youden on scorable rows; sandbox proof as the tiebreaker where a class has recipes; the roadmap rewritten from one command with the unscorable denominator next to every number. +- **Auto-classifier as routing key.** A closed, versioned taxonomy of roughly 8–12 families derived from the CWE hierarchy and aligned to what our verifiers can confirm (sandbox recipe, static corroboration, entry-point and flow tools). Multi-label with an explicit `unknown`. Three tiers: deterministic static signals first, a cheap model second with logprob averaging, `unknown` third, each answer recording its tier. Measured on its own: agreement with maintainer labels on reviewed pairs with hierarchical partial credit, accuracy against a random router and an oracle, confusion matrix per round. Disagreements are the review queue. +- **Per-class detectors, frozen between rounds.** One hunter configuration per family: length-capped prompt, skills, curated tools with symbol resolution (never bash), step and cost budget, evidence checklist, hard negatives, known counterexamples. A region routes to every family the classifier admits plus a generalist. Round zero clones the current hunter into every family so the baseline is controlled; K ≥ 3 runs per class establish the seed-noise floor that becomes that class's regression budget. +- **Class-aware, pair-wise scorer.** Detection = a finding in the labeled function whose class matches the label, with hierarchical credit and the maximum penalty for a hallucinated class. Silence = no finding of the labeled class in the labeled function on the fixed side. Outcomes as pair-correct, both-vulnerable, both-benign, reversed, plus a directional-bias index, negative flips per item against the last blessed round, and a fixed-FPR view. Unscorable rows (identical twin, unsupported language, unresolved label, missing context) listed by reason and removed from the denominator, never deleted. Dedup by normalized hash before splitting; split by repository, chronologically. +- **Only proof is reported above suspicion.** Class-keyed verifiers decide: sandbox recipes, static corroboration, entry-point, flow and obligation tools. A hunter claim without a verifier stays `suspicion` and is scored as such. Verifier code and gold labels sit outside every editable surface, and anything reported as proven gets a second independent judge. +- **Learning unit = structural change, journaled.** A round's output is a tool, a checklist, a counterexample, a memory entry or a bounded prompt edit for one class, drafted from structured failure facts (which sink, which line, which check fired), never from free-form self-critique. Accepted only when train and holdout are both non-negative with one strictly positive and no other class falls below its budget; ties rejected; rejected edits kept in a buffer the proposer sees; the proposer never sees the holdout. HarnessX's `MetaAgent.evolve` and journal (`hypothesis_id`, `levers`, `predicted_affected`, precision discounted by `regressed_unpredicted`) is the round primitive; the tool supplies tasks keyed by class, trajectories, the evaluator and the routing. +- **Honest fitness.** Split enforced by construction; two-stage evaluation (minibatch gate, the class's full holdout, the cross-class sweep); a per-class archive of which candidate wins which instance rather than one champion; the roadmap rewritten from one command with the unscorable denominator and the noise floor beside every number; a per-round cost cap where an over-budget round is reverted and recorded. -Principle for the roadmap: **classify first, then detect, then improve one class at a time under a fitness that punishes what it did not predict.** +Principle for the roadmap: **classify first, verify before reporting, improve one class at a time under a fitness that punishes what it did not predict.** ## Approach -1. **Class-aware scorer and unscorable rows** (repairs the ruler; smallest change, biggest number). Detection and silence keyed on class and labeled function; identical-twin and unsupported-language rows flagged with a reason and excluded from denominators; the hunter's findings carry a class field the scorer reads. Re-publish every slice number from one command. -2. **Taxonomy and auto-classifier.** A closed, versioned class taxonomy (seeded from the 32 free-text labels, the mechanism vocabulary and the obligation kinds; CWE families as the spine). A classifier with two tiers: a deterministic tier from static signals (rule id, sink family, obligation kind, mechanism) and a model tier for what static signals cannot place, each answer carrying which tier decided. Measured against maintainer labels on the reviewed pairs; disagreements are the review queue. The vfc slice's 130 `other` rows are its first job. -3. **Per-class hunter configurations.** Split the single hunter into one configuration per class: prompt, skills directory, tools, budget, evidence checklist. Start by cloning the current hunter into every class so round zero is a controlled baseline, then let rounds diverge them. The classifier routes a target region to its detector; a region can route to several. -4. **Split enforcement and lever refusal.** The exporter and every future lever take the train split by construction and refuse candidates that touch a holdout pair. Re-measure `corpus-seeded-mechanisms` and rewrite its roadmap row. -5. **Evolve loop on HarnessX.** Tasks = classified pairs (`RLTask.task_type` = class); trajectories = the hunter's journaled runs; evaluator = the class-aware scorer; `MetaAgent.evolve` proposes one configuration change per round for one class with `predicted_affected` set to that class; the round is accepted only if the class's holdout improves and no other class regresses beyond tolerance; the journal's attribution table is the round report. Budget-capped per round. -6. **Corpus repairs that the loop needs.** TypeScript grammar in the semantic extra; absence rows re-harvested with `handler_context`; identical twins marked `known_limit` with reason `identical_twin`; a fixture-difficulty check that fails when a unit fixture is strictly easier than the corpus rows its test claims to cover. -7. **Static machinery as tools and verifiers.** Facts, overlay dispositions, obligations, entry points and the sandbox are exposed to the hunter as tools (`entry_points`, `flows`, `obligations`, `prove`) and to the scorer as verifiers that can raise a hunter suspicion to `static_corroboration` or a proof rung. No detector logic is deleted; it changes role. +1. **Class-aware pair scorer and unscorable rows.** Pair-wise outcome vocabulary, location-and-class matching with hierarchical credit, directional-bias index, per-item negative flips, K-run reliable-change test, unscorable reasons out of the denominator, normalized-hash dedup, repository-level split. Re-publish every slice from one command. Smallest change, biggest number. +2. **Taxonomy and auto-classifier.** Family taxonomy versioned in one file with its CWE mapping and its verifier per family; three-tier classifier; its own measurement report; the vfc slice's 130 `other` rows and the 32 free-text classes are its first job, disagreements queued for review. +3. **Per-class hunter configurations.** Clone the current hunter into every family; curated tools with symbol resolution; length-capped prompts; per-family evidence checklist and hard negatives; the classifier routes a region to every admitted family plus the generalist; findings carry their class, detector and verifier. +4. **Verifiers keyed by class.** Sandbox recipes, static corroboration, entry-point, flow and obligation tools exposed to the hunter and to the scorer; a claim rises above `suspicion` only through a verifier; a second independent judge for anything proven. +5. **Split enforcement and lever refusal.** Every exporter and lever takes the train split by construction and refuses candidates touching a holdout pair; re-measure `corpus-seeded-mechanisms` and rewrite its roadmap row. +6. **Evolve loop on HarnessX.** Tasks = classified pairs (`RLTask.task_type` = family); trajectories from journaled hunter runs; evaluator = the class-aware scorer; one structural change for one class per round with `predicted_affected` set; the acceptance rule and cross-class sweep above; per-class Pareto archive; rejected-edit buffer; reflector blind to the holdout; per-round cost cap; the journal's attribution table is the round report and the daily benchmark. +7. **Corpus repairs the loop needs.** TypeScript grammar in the semantic extra; absence rows re-harvested with `handler_context`; identical twins marked `known_limit:identical_twin`; a fixture-difficulty check that fails when a unit fixture is strictly easier than the corpus rows its test claims to cover; a post-cutoff slice for contamination. +8. **Operational floor.** Persisted logs and trajectories, budget caps tested on the failure path, redaction on every prompt and journaled trajectory, cost per point published per round. ## Scope -- **In**: class-aware scorer; taxonomy and classifier with its own measurement; per-class hunter configurations; split enforcement; HarnessX evolve loop wiring with tasks, trajectories, evaluator and journal; the corpus repairs listed in 6; static machinery exposed as hunter tools; roadmap rewrite from one command. -- **Out**: new static shape families; new hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights (skills and configurations only). +- **In**: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6. +- **Out**: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights. ## Boundary Candidates -- Taxonomy and classifier (new, owns class assignment) vs pair labels (unchanged, the reference the classifier is measured against) -- Per-class detector configurations (new, evolved) vs static analyzers (unchanged internally, exposed as tools) -- Evolve loop (new, HarnessX) vs existing bounded levers (rules, policy, mechanisms; kept, subordinated to the split and the class attribution) -- Class-aware scorer (replaces the text-token rule) vs catalog schema (one additive `unscorable` reason) +- Taxonomy and classifier (new, owns class assignment and its measurement) vs pair labels (maintainer-owned reference) +- Per-class detector configurations (new, evolved, frozen between rounds) vs static analyzers (unchanged internally, exposed as tools and verifiers) +- Class-aware scorer (replaces the text-token rule) vs catalog schema (one additive `unscorable` reason, one `class` field) +- Evolve loop (new, HarnessX) vs existing bounded levers (rules, policy, mechanisms; kept, subordinated to split, class and budget) ## Out of Boundary - Taint semantics, walker, dominance internals (`guard-dominance-regime`, `overlay-ir-completeness`) - Call graph and path records (`reachability-flow-model`), except as a future hunter tool -- Obligation checker internals (`authorization-obligations`); its corpus tasks 4.3 and 6.1 move here, its lever task 5 waits for the split +- Obligation checker internals (`authorization-obligations`); its corpus tasks move here, its lever waits for the split ## Upstream / Downstream -- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext` seam, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`. -- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry a class and the detector that produced them). +- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes. +- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier). ## Existing Spec Touchpoints -- **Extends**: `pair-corpus-honesty` Req 1 (scorer rule) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `harnessx-self-improving-rulesets` (the evolve loop lands here). -- **Folds in**: the `closed-loop-integrity` brief (its five repairs are approach items 1, 4 and 6). +- **Extends**: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here). +- **Folds in**: the retired `closed-loop-integrity` brief. - **Re-homes**: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5. ## Constraints -Core `dependencies = []`; HarnessX and grammars stay optional extras; every loop step degrades to a recorded reason without them. No row deleted; unscorable rows keep a reason. Labels stay maintainer-owned: the classifier proposes, disagreements queue for review, and a class change on a reviewed pair is a reviewed edit. Model-authored skills are retained only after held-out validation and are journaled with their attribution; nothing a round proposes reaches a scan without passing its class holdout and the cross-class regression sweep. Split enforced by construction, never by convention. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical: the local slice is not touched by any of this. Redaction applies to every prompt and every journaled trajectory. Per-round cost cap; a round that exceeds it is recorded and reverted. +Core `dependencies = []`; HarnessX and grammars stay optional extras; every loop step degrades to a recorded reason without them. No row deleted; unscorable rows keep a reason. Labels stay maintainer-owned: the classifier proposes, disagreements queue for review, and a class change on a reviewed pair is a reviewed edit. Model-authored skills, tools and prompt edits are retained only after held-out validation under the acceptance rule and are journaled with attribution; nothing a round proposes reaches a scan without its class holdout and the cross-class sweep. Split enforced by construction. Verifier code and gold labels outside every editable surface. Prompt length capped per class. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical. Redaction on every prompt and trajectory. Per-round cost cap with revert. ## Open Questions for the maintainer -1. Taxonomy spine: CWE families, the mechanism vocabulary, or a new closed list with both as attributes? -2. Round zero: clone the current hunter into every class (controlled baseline) or start each class from a hand-written seed prompt? -3. Cross-class regression tolerance: zero, or a per-class floor proportional to class size? -4. Does `authorization-obligations` task 5 (the lever) stay a spec task or become one class's evolve round? -5. Model for the detectors and for the meta-agent: same model, or a cheaper detector with a stronger meta-agent? +1. Taxonomy spine: CWE-699 families aligned to our verifiers (recommended, ~10 families) or the mechanism vocabulary with CWE as an attribute? +2. Regression budget: the measured per-class seed-noise floor (recommended) or a fixed number? +3. Which model runs the detectors and which the meta-agent? Research supports a cheap meta-agent and spending on the detector. +4. Round-zero verifiers: which families get a sandbox recipe first, and do families without a verifier report at `suspicion` only until they have one? +5. Does `authorization-obligations` task 5 (the lever) become one family's evolve round, or stay a spec task? ## Evidence -- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-sonnet5.json`: `ousast pairs --slice vibe-py --split holdout --hunter --hunter-model anthropic/claude-sonnet-5 --json` (2026-09-06). -- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json`: every hunter finding per side for the 17 holdout pairs with `in_labeled_function` and the scorer's verdict; the class-aware re-score uses a keyword classifier written for this measurement only and is the reason approach item 2 exists. +- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-sonnet5.json`: `ousast pairs --slice vibe-py --split holdout --hunter --hunter-model anthropic/claude-sonnet-5 --json`. +- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json`: every hunter finding per side with `in_labeled_function` and the scorer's verdict; the class-aware re-score used a keyword classifier written for the measurement only, which is why approach item 2 exists. +- `research.md`: the four research threads with sources. diff --git a/.kiro/specs/learning-harness/research.md b/.kiro/specs/learning-harness/research.md new file mode 100644 index 0000000..82379e4 --- /dev/null +++ b/.kiro/specs/learning-harness/research.md @@ -0,0 +1,51 @@ +# Research: learning-harness + +Web research conducted 2026-09-06 in four threads by research subagents; summaries are theirs, condensed, with primary sources. Speculation is marked. This file backs the "What others learned" section of `brief.md`. + +## 1. Cyber reasoning systems (AIxCC finals 2025, Big Sleep, XBOW, Glasswing) + +Primary cross-team source: USENIX Security '26 SoK on AIxCC, arXiv 2602.07666. 42-b3yond-6ug and Lacrosse have no first-party write-up. + +| System | Decomposition / routing key | LLM vs tools | +|---|---|---| +| Atlantis (1st), arXiv 2509.14589, team-atlanta.github.io/blog/post-afc | N-version sub-CRSs by language (C, Java, Multilang) with orthogonal designs; six patch agents in parallel, no routing ("no single agent consistently outperformed"); rate limit per fuzzer×sanitizer; Java sinks keyed by Jazzer sanitizer class | Three tiers: LLM-augmented (seeds), LLM-opinionated (hints as optimizations, never correctness), LLM-driven (multi-agent call-graph, bug candidates, blobs). Fuzzers/concolic are ground truth | +| Buttercup (2nd), blog.trailofbits.com 2025-08-08/09 | Seven patch agents by role (RCA, retrieval, writer, reflection); language tooling and CWE guidance in prompts | Fuzzers discover; tree-sitter/CodeQuery program model; cheap non-reasoning LLMs only ($21k for 100k calls); >90% accuracy | +| RoboDuck (3rd), theori.io/blog/aixcc-and-roboduck-63447, /building-effective-llm-agents-63446 | Bug candidate is the anchor: Infer + LLM static analysis + crashes → logprob-scored binary classifier (~$0.001/report) → top 20% to a source-browsing agent (~$0.50) | LLM-first PoVs without fuzzing; three models race, first verified wins; Infer ~99.9% FP | +| FuzzingBrain (4th), arXiv 2509.07225 | 23 strategies keyed on mode × language × stage; per-language CWE lists aligned to sanitizers | LLM-dominant; libFuzzer only; nothing reported unless a sanitizer fires on a real build | +| ARTIPHISHELL (5th), support.shellphish.net 2025-08-22 postmortem | 53 components; multi-source voting; dual patchers | PoV submitted only after five consecutive local reproductions | +| Big Sleep / Naptime, projectzero.google 2024/06 and 2024/10 | No classifier; variant analysis seeded from recent commits | Single agent with code browser, Python, debugger, reporter; must reproduce a crash; parallel independent trajectories | +| XBOW, xbow.com/blog/top-1-how-xbow-did-it | Coordinator spawns per-vulnerability-class solvers; validators keyed by class (canary, headless browser for XSS, timing for blind SQLi) | The non-AI validator "is the only one who can decide if the agent succeeded" | +| Anthropic Glasswing, anthropic.com/research/glasswing-initial-update | Threat-model builder prioritizes targets; scanning subagents; reusable skills | 90.6% of 1,752 findings valid after human reproduction; internals unpublished | + +Lessons: (1) route on mode × language × sanitizer/CWE family, never fine CWE; route to what the verifier can check. (2) Only proof survives; the accuracy multiplier (Atlantis 0.9999 vs 0.9044) decided the ranking; RoboDuck lost 16.3 points to FPs. (3) Cheap classifier → expensive agent → verifier (RoboDuck 500× cost tiering; logprob averaging for stable scores). (4) Ensembles beat selection when the winner is unpredictable; attribution per agent done by hand post hoc. (5) Dedup granularity is a scoring bug (Shadowsocks: five same-pattern bugs differing by line). (6) Validated ≠ correct: 38–46% of automatically accepted patches were semantically wrong (SoK KF4). (7) Hallucination controls are structural: forced tool calls, curated tools instead of bash, symbol resolution via Joern/LSP. (8) Stability beat cleverness: budget caps, cached container state, exit-code-0 logging, lost logs sank teams. (9) Cross-run learning nearly absent; Atlantis's only trained component (GRPO retrieval, C only) showed catastrophic forgetting across projects; daily leakage-resistant CI benchmarks instead. (10) Cost is a control: >$1,000/hour unbounded (arXiv 2603.08566); Buttercup $181/point. + +## 2. Self-improving harness methods + +| Method | Evolved unit | Search operator | Validation | +|---|---|---|---| +| DSPy MIPROv2 (dspy.ai) | Few-shot demos + instructions per module | Bootstrapped traces; LLM proposals; Bayesian optimization | Minibatch trials, full validation every 5 steps; ≥200 examples advised | +| GEPA, arXiv 2507.19457 | One module's prompt per iteration | Natural-language reflection on trajectories; system-aware merge | Minibatch gate → full per-instance Pareto front; candidates sampled ∝ instances led | +| TextGrad; PrefPO critique arXiv 2603.19311 | Prompt text | Textual gradients | Validation revert; 86% of TextGrad prompts hacked/brittle, 14.7× length inflation | +| OPRO 2309.03409 / EvoPrompt 2309.08532 / PromptBreeder 2309.16797 | Task prompt | Proposal / GA / self-referential mutation | Train-score selection; EvoPrompt degrades to generic prose on structured tasks | +| AlphaEvolve (DeepMind blog 2025) | Code diffs | LLM mutation over MAP-Elites program DB | Evaluator cascade; needs automatically verifiable problems | +| Darwin Gödel Machine, arXiv 2505.22954, sakana.ai/dgm | Agent code/tools | Archive sampling ∝ score, ∝ 1/(1+children) | Staged 10→60→200 tasks; held-out models/languages; faked test logs when checker visible | +| ADAS, arXiv 2408.08435 | Agent code | Meta-agent over archive | Held-out validation and domains | +| Reflexion / ExpeL 2308.10144 / AWM 2409.07429 / Voyager 2305.16291 | Reflections / insights / workflows / skills | Self-critique; contrast; induction | Mostly none | +| Self-Harness, arXiv 2606.09498 | Harness components | Weakness mining → K minimal proposals | Accept iff Δ_in ≥ 0 ∧ Δ_ho ≥ 0 ∧ one > 0 | +| SkillOpt, arXiv 2605.23904 | SKILL.md | Reflection with rejected-edit buffer | Strict improvement on held-out; ties rejected | +| AHE, arXiv 2604.25850 | Tools, middleware, memory, prompts | Evidence-driven edits with per-edit manifest (predicted fixes and at-risk regressions) | Prediction vs observed deltas; rollback | +| Meta-Harness, arXiv 2603.28052 | Harness code | Agentic proposer over prior candidates and traces | Held-out models and tasks | + +Lessons: (1) per-instance/per-class tracking doubled gains vs greedy best-overall (GEPA +12.44% vs +6.05%). (2) No-regression acceptance rule; reject ties (Self-Harness, SkillOpt). (3) Regression blindness: AHE fix-prediction precision 33.7% but regression prediction 11.8%; the cross-class sweep is mandatory. (4) Two-stage evaluation is where sample efficiency comes from; most budget is validation. (5) Prompt bloat is the overfit signature (Decagon: +75% length, −2% accuracy; cap ~1,500 chars gave 4× compression at −0.8%). (6) Keep the reflector blind to validation; keep a rejected-edit buffer (arXiv 2602.22483; SkillOpt). (7) Structural edits transfer (+5.1–10.1 points across model families), system-prompt-only edits regressed −2.3 (AHE ablation). (8) Self-diagnosis confabulates: 0/121 Reflexion reflections correct; structured trajectory facts → 86% (arXiv 2605.29463). (9) Hide the checker (DGM). (10) One objective per round (arXiv 2605.26046: joint criteria −59% task focus). (11) Audit that a skill beats re-showing the failure (ContinualSkillBench arXiv 2608.03874). (12) Harness-edit quality flat across model tiers; spend on the detector (arXiv 2605.30621). Speculation: no source measures class-to-class oscillation; per-class Pareto retention plus the acceptance rule should bound it. Anthropic's Claude Code postmortem (InfoQ 2026-05: three unattributed harness changes → six-week degradation) is the cautionary tale. + +## 3. Dataset pitfalls and class taxonomies + +Scoring: (1) dedup by normalized body hash before splitting; split by commit, chronologically (PrimeVul arXiv 2403.18624; test-set copies 12.7–18.9% in older sets). (2) Automatic labels 25–60% correct (BigVul 25%, CVEfixes 51.7%, DiverseVul 60%; Croft et al. ICSE'23 20–71% wrong, 17–99% duplicates; CleanVul arXiv 2411.17274). (3) Pair-wise outcomes P-C/P-V/P-B/P-R, not F1 (GPT-4 CoT P-C 12.94% vs P-V 54.26%; JitVul ACL 2025: agents lose 8–9 points of pair accuracy while "winning" F1; Vul-RAG pairwise 0.21 vs plain 0.61). (4) Directional-bias index (CWE-Trace arXiv 2606.20502: −85.5 to +94.8 points at ~50% accuracy). (5) Score at fixed FPR (PrimeVul VD-S; VFC benchmark arXiv 2605.13138). (6) Match on location and class with hierarchical credit and hallucination penalty (CASTLE arXiv 2503.09433; ALPHA arXiv 2601.01320). (7) Robustness to renames/whitespace/dead code (SecLLMHolmes arXiv 2312.12575; patched-code FPs dominate). (8) CVE-date contamination checks are weak evidence, keep a post-cutoff slice anyway (thin: one paper). (9) Function-level pairs lose context: up to 54% of patched-code misclassification (arXiv 2504.13474). (10) Run-to-run consistency 8.3–81.9% by model (ALPHA). + +Taxonomy: (1) fine CWE from code unsolved (top-1 ≤ 14.7% over 74 CWEs; 86.5% lateral errors; VulDetectBench arXiv 2406.07595 >80% coarse, <30% detailed). (2) From CVE text ~85–90% ceiling (arXiv 2603.14911; CweAgent arXiv 2608.21977). (3) Human/NVD disagreement 15–50%; CWE-699 families as "defensible" (Seclometry; Sonnet vs NVD 73.1% exact, 84.5% with parent–child, LLM right in 72/100 disagreements). (4) Score hierarchically. (5) Few aggregated classes: MoEVD arXiv 2501.16454 collapsed 88 → 12 families, per-class experts +12.8% F1, router 63.8% right, F1 0.18 when wrong; three-class OOB/UAF/other 87.4% (arXiv 2509.22796). (6) Specialists need hard negatives (arXiv 2408.02329: F1 0.89 in class → 0.12–0.19 on full corpus). (7) Per-class detectors work when the class fixes sources/sinks (IRIS arXiv 2405.17238: 55 vs 27 CodeQL detections, FDR 46–85%). (8) VFC typing immature (F1 0.53–0.70). Speculation: 8–12 hierarchy-derived families with partial credit beats fine CWEs for this corpus. + +## 4. Routing and regression control + +Router: (1) learned routers plateau 10–20 points below oracle, sometimes lose to random (RouteLLM arXiv 2406.18665; RouterBench arXiv 2403.12031; arXiv 2606.07587). (2) Evaluate router AUROC separately from expert strength (arXiv 2602.11877). (3) Small embedding classifier fine-tuned on own labels: 80.4% → 98.5% (Red Hat vLLM Semantic Router, 2026-06). (4) Cascades tolerate router error until judge error > 0.2 (FrugalGPT arXiv 2305.05176; RouterBench). (5) Multi-expert routing with balance (Expert-Choice, Google; AgentRouter arXiv 2510.05445); speculation: multi-label with `unknown` is safer than argmax for SAST. + +Attribution: (6) slice everything (Oakden-Rayner arXiv 1909.12475; Slice-based learning 1909.06349; Overton 1909.05372; Model Cards 1810.03993). (7) Negative flips per item (PCT arXiv 2011.09161: 5.2% NFR at equal accuracy; MUSCLE 2407.09435: 8–49% NFR across Llama versions; arXiv 2604.27405: +1.6–2.8 point gains masked 28–39% items worse, single-shot eval missed 42% of changes → K ≥ 3 samples and a reliable-change test; UpgradeBench 2608.20918: neutral retraining breaks 1–5%). (8) Frozen per-class configurations (+20%, Progressive Prompts arXiv 2301.12314); self-feedback-only skill loops drift, external feedback does not (SkillLearnBench 2604.20087); self-revision flips correct → incorrect more often than the reverse (arXiv 2310.01798). (9) Per-instance archives over one champion (GEPA; DGM lineage logs caught faked tests). (10) Conjunctive per-slice gates with relative thresholds vs the last blessed baseline (TFMA model validations; ease.ml/ci arXiv 1903.00278; Amazon Alexa regression-free updates). No public numeric NFR budget; speculation: the measured seed-noise floor per class is the defensible budget.