docs(learning-harness): fold four research threads into the brief and add research.md

AIxCC cyber reasoning systems, self-improving harness methods (GEPA, DGM,
Self-Harness, SkillOpt, AHE), vulnerability-dataset pitfalls and CWE taxonomies,
and routing/regression-control literature. The design changes they force:
coarse verifier-aligned families instead of fine CWE, multi-label routing with
an unknown class, pair-wise outcomes with a directional-bias index, K-run
reliable-change tests, a per-class seed-noise floor as the regression budget,
a no-regression acceptance rule with ties rejected, structural edits over prose
edits, a proposer blind to the holdout, and verifier code and gold labels
outside every editable surface.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

.kiro/specs/learning-harness/brief.md

@@ -11,80 +11,95 @@ Measured 2026-09-06 on the vibe-py holdout (17 vendored pairs), the existing too
 | tool hunter | 0/17 correct, Youden −0.588 | **12/14 correct, Youden +0.857, 0 leaks** |
 | overlay (facts + shapes) | 5/17 correct, Youden +0.294 | 5/14 correct, Youden +0.357 |
 
-The published hunter number is a scoring artifact: the pair scorer counts a hunter finding only when the finding text contains the literal CWE id and the prompt never asks for one, so 24 of 25 vulnerable-side findings that sit inside the labeled function and name the labeled bug (SQL injection in `resolve_pastes`, BOLA in `get_by_title`, SSTI in `read_root`, missing authorization on the profile lookup) were thrown away. The 24 "leaks" split into 3 findings on byte-identical twins (unscorable by construction) and 21 findings outside the labeled function on the trap twin, most of them real defects in a deliberately vulnerable repository. The two remaining misses are `before_request` (`loads(b64decode(cookie))` with no import in the excerpt) and `bad_mvc.do_GET` (labeled CWE-79, the hunter reported the SQL concatenation on the same lines; the label may be wrong).
+The published hunter number is a scoring artifact: the pair scorer counts a hunter finding only when the finding text contains the literal CWE id and the prompt never asks for one, so 24 of 25 vulnerable-side findings that sit inside the labeled function and name the labeled bug were thrown away. The 24 "leaks" split into 3 findings on byte-identical twins and 21 findings outside the labeled function on the trap twin, most of them real defects in a deliberately vulnerable repository. Raw runs: `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-*.json`.
 
 Three consequences follow.
 
-1. **The model with tools is already the best detector we own, by a wide margin, without any learning.** The roadmap thesis ("recall from the model with tools plus proof; precision and silence from static checks") is confirmed, and the harness has been optimizing the wrong component.
-2. **Detection, silence and leak are meaningless without a class.** The same finding is a detection in one class, a leak in another, and noise in a third. Every scorer, lever and gate in the tool currently uses text tokens (CWE strings, sink names) as a proxy for class, and the corpus's own class labels are 32 free-text strings across 7 slices, with 130 of 176 vfc rows labeled `other`.
-3. **Improvements without a class dimension oscillate.** A change that helps SQL injection and hurts access control reads as "no change" on an aggregate Youden. The loop then reverts it, or keeps it, for the wrong reason, and the system degrades in ways nobody predicted. This is the maintainer's observation and it is exactly what the corpus-seeded lever does today.
+1. **The model with tools is already the best detector we own, without any learning.** The roadmap thesis ("recall from the model with tools plus proof; precision and silence from static checks") is confirmed, and the harness has been optimizing the wrong component.
+2. **Detection, silence and leak are meaningless without a class.** Every scorer, lever and gate uses text tokens (CWE strings, sink names) as a proxy for class; the corpus's own labels are 32 free-text strings across 7 slices, and 130 of 176 vfc rows say `other`.
+3. **Improvements without a class dimension oscillate.** A change that helps one class and hurts another reads as no change on an aggregate Youden; the loop keeps or reverts it for the wrong reason and the system degrades in ways nobody predicted. This is the maintainer's observation and it is what the corpus-seeded lever does today.
 
-Repairs already measured and required regardless (from the folded `closed-loop-integrity` brief): the mechanism lever seeds candidates from holdout pairs (holdout Youden +0.059 with every pair teaching, −0.059 with the train split only); five vibe-py twins are byte-identical; TypeScript has no grammar; absence excerpts carry no registration context; unit fixtures are easier than the corpus rows they stand in for.
+Repairs already measured and required regardless: the mechanism lever seeds candidates from holdout pairs (holdout Youden +0.059 with every pair teaching, −0.059 with the train split only); five vibe-py twins are byte-identical; TypeScript has no grammar; absence excerpts carry no registration context; unit fixtures are easier than the corpus rows they stand in for.
+
+## What others learned
+
+Full sourced notes in `research.md` (four threads, 2026-09-06). The findings that change this design:
+
+**Cyber reasoning systems (AIxCC finals, Big Sleep, XBOW, Glasswing).** The routing key that worked in every finalist was mode × language × sanitizer or CWE family, aligned to what the oracle can confirm; nobody routed on fine CWE. Only proof survives: every top system gates a report on an executed oracle, the accuracy multiplier decided the ranking (Atlantis 0.9999 versus 0.9044), and XBOW keys a non-AI validator per vulnerability class that alone decides success. RoboDuck's cheap logprob classifier (~$0.001) sending the top 20% to a $0.50 agent is the closest public analogue of an auto-classifier front end. Ensembles beat selection when the winner is unpredictable, with attribution done per agent after the fact. Validated is not correct: 38–46% of automatically accepted patches were semantically wrong. Hallucination controls are structural (forced tool calls, curated tools instead of bash, symbol resolution). Cross-run learning is nearly absent in the field, and the one trained component reported catastrophic forgetting across projects. Stability and cost decided more than cleverness.
+
+**Self-improving harnesses (GEPA, DSPy, AlphaEvolve, DGM, Self-Harness, SkillOpt, AHE, Meta-Harness).** Track wins per instance or class, never the aggregate: GEPA's per-instance Pareto archive doubled the gain of greedy selection. Accept a change only when train and holdout are both non-negative and one is strictly positive; reject ties. Meta-agents predict the class they help at 33.7% precision and the class they hurt at 11.8% ("regression blindness"), so a cross-class sweep cannot be optional. Two-stage evaluation is where sample efficiency comes from. Prompt bloat is the overfit signature; cap length. Structural edits (tools, checklists, memory) transfer across model families, prose edits regress. Self-diagnosis confabulates (0 of 121 reflections named the real cause; structured trajectory facts reached 86%). Hide the checker and the gold labels from the editable surface. One change, one class per round. Harness-edit quality is flat across model tiers while detector quality is not: a cheap meta-agent, spend on the detector.
+
+**Datasets and taxonomies.** Deduplicate by normalized hash before splitting and split by repository, chronologically. Report pair-wise outcomes (pair-correct, both-vulnerable, both-benign, reversed) and a directional-bias index, not F1; models that flag both sides win F1 while losing pair accuracy. Match on location and class with hierarchical partial credit and the maximum penalty for a hallucinated class. Fine CWE classification from code is unsolved (top-1 ≤ 14.7%, most errors lateral siblings); coarse families are workable (88 CWEs collapsed to 12 gave per-class experts +12.8% F1, with a router right 63.8% of the time and F1 0.18 when wrong). Specialists collapse without hard negatives. Verdicts flip on renames and whitespace; single-shot evaluation missed 42% of real changes, so evaluate with K ≥ 3 runs and a reliable-change test. Function-level pairs lose context, the same defect as our context-free absence excerpts.
+
+**Routing and regression control.** Learned routers plateau 10–20 points below an oracle and can lose to random; measure router accuracy on its own against random and oracle. Route to several experts with an explicit `unknown`, never argmax. Slice everything; aggregate gains of +2 points have masked 28–39% of items getting worse. Freeze per-class configurations to prevent interference; skill loops driven by self-feedback alone drift, external feedback does not. Production gates are conjunctive per slice with relative thresholds against the last blessed baseline; no public numeric negative-flip budget exists, so the defensible budget is the measured seed-noise floor per class.
 
 ## Desired Outcome
 
-A harness that improves a **set of classified detectors** round by round, where every change is attributed to the class it was meant to move, every unpredicted regression in another class counts against it, and nothing is published without its unscorable denominator.
+A harness that improves a **set of classified detectors** round by round, where every change is attributed to the class it was meant to move, every unpredicted regression in another class counts against it, only what a verifier can confirm is reported above suspicion, and nothing is published without its unscorable denominator and its noise floor.
 
-- **Auto-classifier.** Every unit of work (a labeled pair, a finding, a scan target region) is assigned to a class from a closed, versioned taxonomy by a classifier that is itself measured: agreement with maintainer labels on the reviewed pairs, confusion matrix per round. The classifier is the routing key for everything downstream: which detector runs, which skills load, which holdout the change is validated on, which journal row records it.
-- **Per-class detectors.** One hunter configuration per class: system prompt, skills, tool set, step budget, evidence checklist, known counterexamples. A round evolves one class's configuration and is validated on that class's holdout plus a regression sweep over every other class. The static machinery (facts, overlay, obligations, dominance, sandbox) becomes tools the detector calls and verifiers that check its claims; it stops being the detector.
-- **Class-aware scorer.** Detection = a finding in the labeled function whose class matches the label. Silence = no finding of the labeled class in the labeled function on the fixed side. Unscorable rows (identical twin, unsupported language, unresolved label) are listed by reason and removed from the denominator, never deleted. The text-token proxy goes away.
-- **Learning unit = skill, not regex.** A round's output is a natural-language skill, a tool recipe, a checklist or a counterexample, drafted from a failure analysis, tested on the class's train split, validated on its holdout, journaled with a hypothesis, the predicted affected class, and the measured attribution. HarnessX's `MetaAgent.evolve` with its journal (`hypothesis_id`, `levers`, `predicted_affected`, precision discounted by `regressed_unpredicted`) is the loop primitive; the tool supplies tasks, trajectories, the evaluator and the class routing.
-- **Honest fitness.** Split enforced by construction (a candidate touching a holdout pair is refused, not scored); per-class Youden on scorable rows; sandbox proof as the tiebreaker where a class has recipes; the roadmap rewritten from one command with the unscorable denominator next to every number.
+- **Auto-classifier as routing key.** A closed, versioned taxonomy of roughly 8–12 families derived from the CWE hierarchy and aligned to what our verifiers can confirm (sandbox recipe, static corroboration, entry-point and flow tools). Multi-label with an explicit `unknown`. Three tiers: deterministic static signals first, a cheap model second with logprob averaging, `unknown` third, each answer recording its tier. Measured on its own: agreement with maintainer labels on reviewed pairs with hierarchical partial credit, accuracy against a random router and an oracle, confusion matrix per round. Disagreements are the review queue.
+- **Per-class detectors, frozen between rounds.** One hunter configuration per family: length-capped prompt, skills, curated tools with symbol resolution (never bash), step and cost budget, evidence checklist, hard negatives, known counterexamples. A region routes to every family the classifier admits plus a generalist. Round zero clones the current hunter into every family so the baseline is controlled; K ≥ 3 runs per class establish the seed-noise floor that becomes that class's regression budget.
+- **Class-aware, pair-wise scorer.** Detection = a finding in the labeled function whose class matches the label, with hierarchical credit and the maximum penalty for a hallucinated class. Silence = no finding of the labeled class in the labeled function on the fixed side. Outcomes as pair-correct, both-vulnerable, both-benign, reversed, plus a directional-bias index, negative flips per item against the last blessed round, and a fixed-FPR view. Unscorable rows (identical twin, unsupported language, unresolved label, missing context) listed by reason and removed from the denominator, never deleted. Dedup by normalized hash before splitting; split by repository, chronologically.
+- **Only proof is reported above suspicion.** Class-keyed verifiers decide: sandbox recipes, static corroboration, entry-point, flow and obligation tools. A hunter claim without a verifier stays `suspicion` and is scored as such. Verifier code and gold labels sit outside every editable surface, and anything reported as proven gets a second independent judge.
+- **Learning unit = structural change, journaled.** A round's output is a tool, a checklist, a counterexample, a memory entry or a bounded prompt edit for one class, drafted from structured failure facts (which sink, which line, which check fired), never from free-form self-critique. Accepted only when train and holdout are both non-negative with one strictly positive and no other class falls below its budget; ties rejected; rejected edits kept in a buffer the proposer sees; the proposer never sees the holdout. HarnessX's `MetaAgent.evolve` and journal (`hypothesis_id`, `levers`, `predicted_affected`, precision discounted by `regressed_unpredicted`) is the round primitive; the tool supplies tasks keyed by class, trajectories, the evaluator and the routing.
+- **Honest fitness.** Split enforced by construction; two-stage evaluation (minibatch gate, the class's full holdout, the cross-class sweep); a per-class archive of which candidate wins which instance rather than one champion; the roadmap rewritten from one command with the unscorable denominator and the noise floor beside every number; a per-round cost cap where an over-budget round is reverted and recorded.
 
-Principle for the roadmap: **classify first, then detect, then improve one class at a time under a fitness that punishes what it did not predict.**
+Principle for the roadmap: **classify first, verify before reporting, improve one class at a time under a fitness that punishes what it did not predict.**
 
 ## Approach
 
-1. **Class-aware scorer and unscorable rows** (repairs the ruler; smallest change, biggest number). Detection and silence keyed on class and labeled function; identical-twin and unsupported-language rows flagged with a reason and excluded from denominators; the hunter's findings carry a class field the scorer reads. Re-publish every slice number from one command.
-2. **Taxonomy and auto-classifier.** A closed, versioned class taxonomy (seeded from the 32 free-text labels, the mechanism vocabulary and the obligation kinds; CWE families as the spine). A classifier with two tiers: a deterministic tier from static signals (rule id, sink family, obligation kind, mechanism) and a model tier for what static signals cannot place, each answer carrying which tier decided. Measured against maintainer labels on the reviewed pairs; disagreements are the review queue. The vfc slice's 130 `other` rows are its first job.
-3. **Per-class hunter configurations.** Split the single hunter into one configuration per class: prompt, skills directory, tools, budget, evidence checklist. Start by cloning the current hunter into every class so round zero is a controlled baseline, then let rounds diverge them. The classifier routes a target region to its detector; a region can route to several.
-4. **Split enforcement and lever refusal.** The exporter and every future lever take the train split by construction and refuse candidates that touch a holdout pair. Re-measure `corpus-seeded-mechanisms` and rewrite its roadmap row.
-5. **Evolve loop on HarnessX.** Tasks = classified pairs (`RLTask.task_type` = class); trajectories = the hunter's journaled runs; evaluator = the class-aware scorer; `MetaAgent.evolve` proposes one configuration change per round for one class with `predicted_affected` set to that class; the round is accepted only if the class's holdout improves and no other class regresses beyond tolerance; the journal's attribution table is the round report. Budget-capped per round.
-6. **Corpus repairs that the loop needs.** TypeScript grammar in the semantic extra; absence rows re-harvested with `handler_context`; identical twins marked `known_limit` with reason `identical_twin`; a fixture-difficulty check that fails when a unit fixture is strictly easier than the corpus rows its test claims to cover.
-7. **Static machinery as tools and verifiers.** Facts, overlay dispositions, obligations, entry points and the sandbox are exposed to the hunter as tools (`entry_points`, `flows`, `obligations`, `prove`) and to the scorer as verifiers that can raise a hunter suspicion to `static_corroboration` or a proof rung. No detector logic is deleted; it changes role.
+1. **Class-aware pair scorer and unscorable rows.** Pair-wise outcome vocabulary, location-and-class matching with hierarchical credit, directional-bias index, per-item negative flips, K-run reliable-change test, unscorable reasons out of the denominator, normalized-hash dedup, repository-level split. Re-publish every slice from one command. Smallest change, biggest number.
+2. **Taxonomy and auto-classifier.** Family taxonomy versioned in one file with its CWE mapping and its verifier per family; three-tier classifier; its own measurement report; the vfc slice's 130 `other` rows and the 32 free-text classes are its first job, disagreements queued for review.
+3. **Per-class hunter configurations.** Clone the current hunter into every family; curated tools with symbol resolution; length-capped prompts; per-family evidence checklist and hard negatives; the classifier routes a region to every admitted family plus the generalist; findings carry their class, detector and verifier.
+4. **Verifiers keyed by class.** Sandbox recipes, static corroboration, entry-point, flow and obligation tools exposed to the hunter and to the scorer; a claim rises above `suspicion` only through a verifier; a second independent judge for anything proven.
+5. **Split enforcement and lever refusal.** Every exporter and lever takes the train split by construction and refuses candidates touching a holdout pair; re-measure `corpus-seeded-mechanisms` and rewrite its roadmap row.
+6. **Evolve loop on HarnessX.** Tasks = classified pairs (`RLTask.task_type` = family); trajectories from journaled hunter runs; evaluator = the class-aware scorer; one structural change for one class per round with `predicted_affected` set; the acceptance rule and cross-class sweep above; per-class Pareto archive; rejected-edit buffer; reflector blind to the holdout; per-round cost cap; the journal's attribution table is the round report and the daily benchmark.
+7. **Corpus repairs the loop needs.** TypeScript grammar in the semantic extra; absence rows re-harvested with `handler_context`; identical twins marked `known_limit:identical_twin`; a fixture-difficulty check that fails when a unit fixture is strictly easier than the corpus rows its test claims to cover; a post-cutoff slice for contamination.
+8. **Operational floor.** Persisted logs and trajectories, budget caps tested on the failure path, redaction on every prompt and journaled trajectory, cost per point published per round.
 
 ## Scope
 
-- **In**: class-aware scorer; taxonomy and classifier with its own measurement; per-class hunter configurations; split enforcement; HarnessX evolve loop wiring with tasks, trajectories, evaluator and journal; the corpus repairs listed in 6; static machinery exposed as hunter tools; roadmap rewrite from one command.
-- **Out**: new static shape families; new hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights (skills and configurations only).
+- **In**: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
+- **Out**: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
 
 ## Boundary Candidates
 
-- Taxonomy and classifier (new, owns class assignment) vs pair labels (unchanged, the reference the classifier is measured against)
-- Per-class detector configurations (new, evolved) vs static analyzers (unchanged internally, exposed as tools)
-- Evolve loop (new, HarnessX) vs existing bounded levers (rules, policy, mechanisms; kept, subordinated to the split and the class attribution)
-- Class-aware scorer (replaces the text-token rule) vs catalog schema (one additive `unscorable` reason)
+- Taxonomy and classifier (new, owns class assignment and its measurement) vs pair labels (maintainer-owned reference)
+- Per-class detector configurations (new, evolved, frozen between rounds) vs static analyzers (unchanged internally, exposed as tools and verifiers)
+- Class-aware scorer (replaces the text-token rule) vs catalog schema (one additive `unscorable` reason, one `class` field)
+- Evolve loop (new, HarnessX) vs existing bounded levers (rules, policy, mechanisms; kept, subordinated to split, class and budget)
 
 ## Out of Boundary
 
 - Taint semantics, walker, dominance internals (`guard-dominance-regime`, `overlay-ir-completeness`)
 - Call graph and path records (`reachability-flow-model`), except as a future hunter tool
-- Obligation checker internals (`authorization-obligations`); its corpus tasks 4.3 and 6.1 move here, its lever task 5 waits for the split
+- Obligation checker internals (`authorization-obligations`); its corpus tasks move here, its lever waits for the split
 
 ## Upstream / Downstream
 
-- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext` seam, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`.
-- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry a class and the detector that produced them).
+- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
+- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
 
 ## Existing Spec Touchpoints
 
-- **Extends**: `pair-corpus-honesty` Req 1 (scorer rule) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `harnessx-self-improving-rulesets` (the evolve loop lands here).
-- **Folds in**: the `closed-loop-integrity` brief (its five repairs are approach items 1, 4 and 6).
+- **Extends**: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
+- **Folds in**: the retired `closed-loop-integrity` brief.
 - **Re-homes**: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
 
 ## Constraints
 
-Core `dependencies = []`; HarnessX and grammars stay optional extras; every loop step degrades to a recorded reason without them. No row deleted; unscorable rows keep a reason. Labels stay maintainer-owned: the classifier proposes, disagreements queue for review, and a class change on a reviewed pair is a reviewed edit. Model-authored skills are retained only after held-out validation and are journaled with their attribution; nothing a round proposes reaches a scan without passing its class holdout and the cross-class regression sweep. Split enforced by construction, never by convention. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical: the local slice is not touched by any of this. Redaction applies to every prompt and every journaled trajectory. Per-round cost cap; a round that exceeds it is recorded and reverted.
+Core `dependencies = []`; HarnessX and grammars stay optional extras; every loop step degrades to a recorded reason without them. No row deleted; unscorable rows keep a reason. Labels stay maintainer-owned: the classifier proposes, disagreements queue for review, and a class change on a reviewed pair is a reviewed edit. Model-authored skills, tools and prompt edits are retained only after held-out validation under the acceptance rule and are journaled with attribution; nothing a round proposes reaches a scan without its class holdout and the cross-class sweep. Split enforced by construction. Verifier code and gold labels outside every editable surface. Prompt length capped per class. Gates (`gate`, `map_gate`, `pair_gate`) stay byte-identical. Redaction on every prompt and trajectory. Per-round cost cap with revert.
 
 ## Open Questions for the maintainer
 
-1. Taxonomy spine: CWE families, the mechanism vocabulary, or a new closed list with both as attributes?
-2. Round zero: clone the current hunter into every class (controlled baseline) or start each class from a hand-written seed prompt?
-3. Cross-class regression tolerance: zero, or a per-class floor proportional to class size?
-4. Does `authorization-obligations` task 5 (the lever) stay a spec task or become one class's evolve round?
-5. Model for the detectors and for the meta-agent: same model, or a cheaper detector with a stronger meta-agent?
+1. Taxonomy spine: CWE-699 families aligned to our verifiers (recommended, ~10 families) or the mechanism vocabulary with CWE as an attribute?
+2. Regression budget: the measured per-class seed-noise floor (recommended) or a fixed number?
+3. Which model runs the detectors and which the meta-agent? Research supports a cheap meta-agent and spending on the detector.
+4. Round-zero verifiers: which families get a sandbox recipe first, and do families without a verifier report at `suspicion` only until they have one?
+5. Does `authorization-obligations` task 5 (the lever) become one family's evolve round, or stay a spec task?
 
 ## Evidence
 
-- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-sonnet5.json`: `ousast pairs --slice vibe-py --split holdout --hunter --hunter-model anthropic/claude-sonnet-5 --json` (2026-09-06).
-- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json`: every hunter finding per side for the 17 holdout pairs with `in_labeled_function` and the scorer's verdict; the class-aware re-score uses a keyword classifier written for this measurement only and is the reason approach item 2 exists.
+- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-sonnet5.json`: `ousast pairs --slice vibe-py --split holdout --hunter --hunter-model anthropic/claude-sonnet-5 --json`.
+- `benchmarks/measurements/2026-09-06-vibe-py-holdout-hunter-findings.json`: every hunter finding per side with `in_labeled_function` and the scorer's verdict; the class-aware re-score used a keyword classifier written for the measurement only, which is why approach item 2 exists.
+- `research.md`: the four research threads with sources.