- Ten families exactly as decided, each with its CWE set, mechanism set, verifier kind (canary, static, none) and description; `unknown` always present with no verifier; a version string.
- Loader rejects an unknown field, an eleventh family, a missing verifier kind or a CWE claimed by two families, naming the offender; relation lookup answers same, parent–child, lateral or fabricated for any two ids.
- Observable: the taxonomy loads from the ruleset directory, a fixture with a fabricated family fails by name, and every mechanism id in the mechanism vocabulary resolves to exactly one family.
- Endpoint resolution order: explicit override, scripted client modes, DeepSeek key and base URL, then OpenRouter; the embedding client keeps reading only its own variables; `[models]` gains the hunter and judge defaults and the endpoint keys, `[learning]` gains its section.
- Adapter disables thinking and sends temperature zero for detector and classifier calls, keeps the provider's reasoning field on assistant messages so a caller can replay it, supports JSON-object mode with one retry on empty content and optional log-probabilities, and reports normalized usage and cost from the provider's cache-hit fields.
- Observable: with both provider keys set the hunter client targets the DeepSeek base URL without a version suffix while the embedding client still targets OpenRouter; a scripted client is chosen under the existing environment flag; a recorded request shows thinking disabled and the reasoning field retained on the reply.
- Expected rows accept an opaque family id (validated later by the classifier task, so the corpus module never imports the learning package); a pair carries a computed unscorable reason (identical twin by normalized body comparison with headers stripped, computed only when the catalog does not already declare a known limit) and is never dropped; duplicates across pairs are reported before any split; a repository-and-date split helper assigns train or holdout to rows that lack a split, keeps declared splits, and reports rows from one repository that straddle the split; pair signals take a split filter.
- Observable: the five identical vibe-py twins load with the identical-twin reason and the existing catalogs load with unchanged row counts; the straddle report on the current catalogs is produced and committed.
- The semantic extra gains the TypeScript grammar and the parser maps TypeScript and TSX to it; the extra-free suite still skips them with the existing marker.
- Observable: every vendored TypeScript pair parses on both sides under the extra, and the parser reports no unsupported-language result for TypeScript or TSX.
- Four new repository tools beside the existing three: resolve a definition by symbol, list entry points, list flows for a function, list obligations for a function, each clamped to the scanned repository and returning identifiers and line spans only; the hunter loop gains prompt, context files, tool set, character limit and tag parameters with today's behavior as defaults and replays the provider's reasoning field across tool turns.
- Observable: the existing hunter tests pass unchanged with default parameters; each new tool answers on the shared handler fixture and refuses a path outside the repository; a recorded tool-turn conversation shows the reasoning field replayed.
- Tier one decides from rule id, sink family, obligation kind, mechanism id or labeled CWE without a model; tier two asks the chat endpoint in JSON-object mode with thinking disabled and averages log-probabilities, falling to unknown under a threshold; tier three is unknown; every answer records its tier; results are multi-label; loaded catalog family labels are validated against the taxonomy here and a fabricated family is rejected by name.
- Observable: on the shared handler fixture tier one classifies the leaky handler as access control without any client; with a scripted client an off-vocabulary answer becomes unknown; a catalog with a fabricated family is rejected by name.
- Report: agreement with maintainer labels on reviewed pairs with parent–child partial credit and no credit for lateral or fabricated ids, confusion matrix per family, accuracy against a random router and an oracle router, counts per tier and unknown; disagreements append to a review queue and never change a label; the whole corpus, including free-text and `other` rows, is classified with per-family counts.
- Observable: the report on the vendored corpus lists the vfc `other` rows by resulting family and the queue holds every disagreement once with both answers and the tier.
- Detection only for a finding whose family tag relates same or parent–child to the labeled family and whose line lies in the labeled function, with the maximum penalty for a fabricated family; silence only when no such finding lies in the labeled function on the fixed side; other findings counted separately, never as leaks; text is never inspected; outcomes pair-correct, both-flagged, both-silent, reversed and unscorable; the unscorable reason also covers unsupported language, unresolved label and missing context.
- Observable: a synthetic run set where every fixed side carries an other-family finding scores full silence; a finding with a fabricated family never counts; a pair in an unsupported language scores unscorable with that reason.
- Per family and slice: outcome counts, recall, silence, Youden, directional-bias index, a fixed-false-positive-rate view; K runs per pair with the majority outcome, negative flips per family against a baseline set, and a paired reliable-change test; unscorable rows listed by reason and excluded from every denominator; every metrics block carries the taxonomy version and the scorable and unscorable counts.
- Observable: a single-run difference below the reliable-change threshold reports no change while a consistent flip across runs does; the unscorable count and taxonomy version appear in every metrics block.
- A configuration directory per family: prompt under a stated length cap, checklist, tool list from the curated set only, step and cost budget, maximum characters, hard negatives and counterexamples, version; the loader rejects an over-cap prompt, an unknown tool or a missing family; this task ships the loader and a test fixture configuration, the shipped per-family directories are written by round zero.
- The runner takes the families admitted by the classifier as input, builds the detector's user message from the whole file plus definitions its tools resolve, runs the hunter loop with the family's prompt, tools and budgets, and tags each finding with exactly one family tag and one detector-version tag; the generalist is the unknown family's configuration.
- Observable: a family run on the shared fixture with a scripted client yields findings tagged with that family and version and uses only the family's tool list; a configuration whose prompt exceeds the cap fails to load by name.
- The regress verdict accepts an optional oracle callback and keeps exit-code behavior otherwise; a registry keyed by the taxonomy's verifier kind returns a canary, static or no-op verifier; the static verifier raises access-control claims to static corroboration when the obligations checker reports the same function; the no-op verifier leaves suspicion.
- Observable: with the fake sandbox runner a result whose output carries the token yields triggerable while a nonzero exit without the token does not; a family with no verifier keeps suspicion on the shared fixture; the access-control claim on the leaky handler reaches static corroboration.
- Family snippet templates for injection, file path and deserialization plant a canary (row in an in-memory database, file under the scratch mount, marked object) and print the token only when the vulnerability fires; the second judge asks the judge model with the snippet, oracle output and claim and answers confirm or not in JSON-object mode; a claim is published as proven only when it confirms; templates, oracle code and gold labels live outside every proposer write root.
- Observable: every canary template passes the existing snippet safety check; a scripted judge that declines keeps the claim below proven; templates and oracle code live under the verifier directory and gold labels under the corpus directory.
- Append-only round records with hypothesis, levers, predicted affected and at-risk families, outcome, reason, target deltas, per-family sweep flips, attribution (flipped predicted, flipped unpredicted, precision), cost, taxonomy and configuration versions; a per-family archive of which configuration version wins which pair; a rejected-hypothesis buffer per family; every persisted text passes redaction.
- Observable: a record round-trips through the file; the buffer returns only the target family's rejected hypotheses; the archive names one winner per pair per family; a secret literal written into a record is stored redacted.
- Teachers are vendored, gating-tier, train-split, scorable pairs; a refusal names any candidate taught by or touching a holdout pair; the exporter, leave-one-out, per-profile evaluation, mechanism-profile evaluation, mechanism admission and pair signals all obtain teachers through the rule and record refusals as degradations; the mechanism lever's holdout number is re-measured under the rule and its raw artifact committed for the publish task.
- Observable: leave-one-out on the toy slice seeds from train pairs only; a candidate shape taught by a holdout pair is refused by name; the re-measured artifact is in the measurements directory.
- The pair scorer's hunter path takes an injected family matcher and leak rule from the learning package, classifies each pair once, runs the admitted family detectors plus the generalist K times per side, and emits family outcomes and metrics; the overlay and inventory paths stay untouched.
- Observable: the pairs command with a scripted client on the shared fixture reports a family outcome per pair; no text-token matching remains on the hunter path; the pair gate output is byte-identical to before.
- Clone the current hunter prompt into every family configuration; run each family K = 5 times over the scorable vendored pairs of the selected slices at deterministic settings; record the per-family negative-flip rate between runs as that family's regression budget; persist noise floors and round-zero artifacts keyed by detector model so another model can be baselined and compared without touching configurations; the web families over vibe-py and agent-vfc are the default selection and the memory-safety family over vfc is selectable but never gates.
- Observable: with a scripted client round zero writes a noise-floor file with one budget per family and a round-zero report per slice; a second baseline under another model name writes a separate artifact and the comparison table lists both.
- In a standard scan each classified region runs its admitted family detectors plus the generalist from the directories round zero wrote (a missing family directory is a recorded degradation and that family's detector is skipped), claims pass through the family's verifier, findings carry family, detector and verifier tags in reports and SARIF, and the manifest gains a learning block with taxonomy version, per-family counts, the classifier report and the current round; the detection and map gates stay byte-identical.
- Observable: a standard scan of the shared fixture with a scripted client writes findings tagged with a family, a detector version and a verifier rung, and the manifest block; the detection and map gate outputs diff empty.
- Failure facts for one family: misses and leaks with pair, function, sink or check, line, verifier outcome and run outcomes, plus the rejected buffer and the current configuration; a proposal is one hypothesis, one lever (tool, checklist, counterexample, memory, prompt), one file-level change inside the family directory, and predicted affected and at-risk families; the proposer never receives holdout inputs, outputs or scores.
- A scripted proposer for tests and offline rounds; a HarnessX proposer composing the meta-agent with write roots limited to the family directory, degrading to a recorded reason without the extra.
- Observable: the scripted proposer returns a proposal from a facts fixture; the HarnessX proposer is skipped with the reason recorded when the extra is absent and, when present, writes only inside the family directory; the write-root list excludes the verifier and corpus directories.
- Acceptance requires train and holdout non-negative with one strictly positive and every other family within its budget; ties reject; a family directory snapshot restores byte for byte on rejection or overrun; refusals for holdout contact, length cap and lever set are outcomes with reasons.
- Observable: a helpful change is accepted, a tie is rejected, a change that breaches another family's budget is rejected with the family named, and a revert restores the directory byte for byte.
- One round: facts, proposal, refusals, train minibatch, target-family holdout, sweep over every other family's holdout, K-run scoring at every stage as the scorer requires, accept (freeze, bump version, archive) or revert; cost metered per stage with revert and a recorded reason on overrun; a journal record and the rejected buffer written for every outcome; a round directory holds the snapshot, proposal, scores and trajectories and a repeated round number is refused.
- Observable: with the scripted proposer each of accepted, rejected, budget-breach and cost-overrun leaves one journal record with attribution and a round directory; a repeated round number is refused.
- The round directory holds the pair identities behind the attribution counts and every detector trajectory, redacted, so a round can be replayed and its attribution audited; every stage scores at least three runs per pair.
- One entry point regenerates every family and slice number from committed artifacts, writes a markdown table with scorable and unscorable counts, taxonomy version, noise floor and cost per point beside every number, replaces the roadmap's measured section between markers, and writes the raw artifacts under the measurements directory.
- Observable: running the publish entry point twice without new artifacts produces byte-identical output; the corrected mechanism-lever number from the re-measurement artifact replaces the earlier roadmap value.
- Subcommands for classify, score, baseline, round and publish with the flags the design names; the pairs hunter path gains the K-runs flag; every step runs under the harness runtime with degradations for missing extras, endpoints or sandbox.
- Observable: each subcommand exits zero on the shared fixture with a scripted client and writes its artifact; the roadmap section written by the publish subcommand carries the command line that produced it; a missing endpoint records the reason and the remaining steps complete.
- Absence rows in vibe-py and agent-vfc gain their family and the handler-context mode in the recipes; the maintainer re-harvests them over the network; every re-harvested excerpt keeps its license line, passes redaction and keeps the decorators and registration statements; failed fetches keep the old excerpt with a recorded reason; catalogs regenerate idempotently.
- Observable: the vampi and threatbyte excerpts contain their registration lines, the entry-point mapper names their handlers, and the catalogs load with unchanged vendored counts.
- The five identical vibe-py twins carry the identical-twin known-limit reason in the catalog; a test derives a difficulty vector for a unit fixture (binding hops to the identity source, registration present, cross-file registration) and for the corpus rows its docstring names, and fails when the fixture is strictly easier.
- Observable: the twins are excluded from every denominator with their reason listed; the shared obligations fixture is compared against the re-harvested vampi rows and the check's verdict is recorded.
- Catalog rows gain a fix date populated by the catalog generator from recipe metadata where available; rows without a date are counted separately; rows dated after the configured detector cutoff form a separately reported slice in every scoreboard.
- Observable: the scoreboard shows the post-cutoff slice with its own counts, the cutoff date it used and the undated count.
- Detection, map and local pair gate outputs byte-identical to the pre-feature commit; the extra-free suite green with semantic and HarnessX tests skipping; each optional dependency's absence exercised once with its recorded reason.
- Observable: the three gate outputs diff empty against the recorded baseline and both test matrices pass.
- _Requirements: 11.4, 11.5_
- [ ] 5.2 Live round zero
- Round zero through the DeepSeek endpoint on the vibe-py and agent-vfc web families with K = 5, plus the memory-safety family over the vfc slice as a measured, non-gating run; artifacts, noise floors and cost per pair committed; the classifier report and review queue produced on the same run; a second baseline with the judge model on one slice for the model comparison table.
- Observable: the published table shows per-family outcomes, noise floors and cost for both models and for the vfc memory-safety family, and the roadmap section is regenerated from it.
- **Blocked 2026-09-06: no `DEEPSEEK_API_KEY` in `.env` or the environment.** Everything else is ready; the run is one command per baseline. The maintainer chose to add the key and have the full run go ahead (web families with `deepseek-v4-flash`, the vfc memory slice as a measured non-gating run, and a second baseline with `deepseek-v4-pro` on one slice). Scale, measured on the corpus: 58 scorable web pairs plus 176 vfc pairs, both sides, K=5 = 2,340 detector runs at up to four model calls each; the design estimated under $15 at peak flash pricing.
- Dress-rehearsed offline 2026-09-06 with the scripted client over the whole vibe-py slice at K=3: round zero produced six families with per-slice blocks, correct denominators (the twins out by reason), and a floor each; a round then read the recorded proposals, applied one, swept the other five families for negative flips (0 each), rejected on `tie`, reverted the family directory byte for byte and archived the rejected version's per-pair results. The only thing the live run adds is a detector that can actually see.
- **Amended 2026-09-06 (maintainer, in session): the target family is `injection`, not `access_control`.** Measured on the corpus under the split and review-tier rules: access_control has **1 train teacher and 4 holdout pairs**, six of its rows unusable because every agent-vfc row sits at tier `title`, and the vibe-py absence rows that could teach are Real-Vuln-Benchmark trap pairs whose two sides are different handlers — so the train split teaches **zero** obligation shapes and any accepted change would rest on one pair. `injection` has 12 train teachers and 10 holdout pairs, the only family in the corpus with enough train evidence to tell a real change from noise. The first memory entries are therefore the mechanism shapes the train split really teaches rather than obligation shapes, which do not exist yet. access_control follows when its corpus can teach: either a maintainer reviews agent-vfc rows up to `reviewed`, or the trap rows gain non-trap twins.
- Three rounds with the scripted proposer on the injection family using the shapes the train split teaches as the first memory entries, then one round with the HarnessX proposer; every round journaled with attribution; the roadmap regenerated.
- Observable: the journal holds four records with outcomes and attribution precision, at least one accepted change is versioned in its family directory, and no other family exceeded its budget in any accepted round.
- _Depends: 5.2_
- _Requirements: 9.9_
## Task-plan review notes
- Sanity review round 1 (2026-09-06): NEEDS_FIXES, eleven findings, all repaired (K-run policy honors Req 3.5; per-model baselines added to the design; family validation moved out of the corpus loader; reasoning replay owned by the hunter loop; Req 5.3 split into re-measure and publish; scoring path split into pairs and scan tasks; oversized tasks split; fix-date prerequisite added; fixture-difficulty check depends on the re-harvest; memory-safety family measured on vfc; detector task depends on the classifier and round zero owns the shipped directories; observables re-anchored).
- Sanity review round 2 (2026-09-06): NEEDS_FIXES with three local items, repaired without a third review under the bounded loop: task 1.5 delivers the four curated tools and the hunter-loop parameters that 2.5 relied on; the scan-path task now follows round zero and degrades when a family directory is missing; the write-root observable moved to the HarnessX proposer task with 7.6. Non-blocking notes applied: the split helper assigns as well as reports (4.4), 3.2 depends on 2.1 rather than 2.2.
- Task 1.1 (2026-09-06, RED first: 9 failing tests in `tests/test_learning_families.py`, `ModuleNotFoundError: openultrasast.learning`; then green): `learning/families.py` and `ruleset/families.toml` with the ten decided families, each carrying its CWE set, mechanism set, verifier kind and description. Decisions taken while writing the data: (a) `unknown` carries the six mechanisms that span families (`source_reaches_sink`, `container_taint`, `validation_strength`, `redos`, `cross_artifact`, `other`) and the two weaknesses we deliberately do not target (CWE-20 generic validation, CWE-835 infinite loop), so a row we considered and declined is visible instead of unresolved; an earlier draft said `unknown` had none. (b) The maintainer's "server-side request forgery" family is named `untrusted_destination` and also covers open redirect (CWE-601), because both are untrusted input choosing a request destination and the corpus carries 34 open-redirect rows that would otherwise be unclassifiable. (c) The taxonomy is flat: `parent` is an optional field no shipped family uses, exercised by a fixture, because Req 2.3 and 3.1 need a parent-child arm and inventing a hierarchy would be worse than leaving it empty. (d) Abstention earns no partial credit: `related("unknown", X)` is `lateral`. All 33 labeled corpus weaknesses resolve, none is claimed twice, and every mechanism id belongs to exactly one family.
- Task 1.2 (2026-09-06, RED first: 8 failing tests in `tests/test_learning_endpoint.py`; then green): `learning/endpoint.py` resolves override, then the scripted flag, then a configured endpoint, then DeepSeek, then OpenRouter; the embedding client is untouched. `provider/openrouter.py` gains `complete_chat_raw` (whole payload, for `usage`) and an `extra_body` passthrough; `ChatResponse` gains `reasoning` and `mean_logprob`. Design deviation: `ChatEndpoint` deliberately holds no `api_key` (the design listed one) so the record is safe to journal; the key stays on the client. `[models]` gains `judge`, `chat_base_url`, `chat_api_key_env`; `[learning]` is new. `ModelConfig.hunter` keeps its `None` default on purpose: defaulting it would silently enable LLM stages in existing scans, so the DeepSeek model defaults live in `resolve_models`.
- Task 1.3 (2026-09-06, RED first: 7 failing tests in `tests/test_pair_hygiene.py`; then green): `ExpectedFinding.family` (opaque; the classifier validates it), `PairCase.unscorable`, `PairCase.split_declared`, `PairCase.fix_date` (the generator fills it in 4.3), `duplicate_groups`, `split_by_repository` with `SplitReport`, and a split filter on `build_pair_signals`. Measured and committed as `benchmarks/measurements/2026-09-06-corpus-hygiene.json`: **19 repositories straddle the train/holdout split** (a learning path can be taught by one file of a repository and evaluated on another) and **10 groups of pairs share a vulnerable excerpt byte for byte**, mostly the same fix harvested twice at different granularity. Only three vendored rows (the `local` slice) declare no split.
- Task 1.4 (2026-09-06, RED first: 2 failing tests in `tests/test_typescript_grammar.py`; then green): the semantic extra gains `tree-sitter-typescript`; `grammar_for` maps a language to (module, attribute) because the wheel ships two grammars, and `parse_with_cst` picks the JSX-aware one for a `.tsx` path while the parsed file keeps the language it was given. Eighteen agent-vfc rows plus the TypeScript absence rows now parse on both sides.
- Task 1.5 (2026-09-06, RED first: 5 of 6 tests in `tests/test_hunter_curated_tools.py` failing; then green): `hunter_tools` gains `read_definition`, `entry_points`, `flows`, `obligations`; `run_tool_hunter` gains `system_prompt`, `user_prompt`, `context_files`, `tools`, `max_chars` and `tags`, all defaulting to today's behaviour, and replays the provider's reasoning field on the next turn. Deviations: `read_definition` returns the definition text as well as its span, and includes preceding decorator lines, because a resolver that returns only line numbers forces another `read_file` and a handler's decorators are exactly what the obligations work showed to matter; the three list tools return identifiers and spans only. No caching: a stale view of a tree a round may have rewritten is worse than a reparse.
- Review round 1 (2026-09-06) APPROVED 1.1, 1.2, 1.4, 1.5; REJECTED 1.3. The reviewer ran every check in a detached worktree at the group-1 tip and confirmed RED-first by running each new test file against the pre-feature commit (37 failures there, all passing after), the three gate outputs byte-identical, and `pairs`/`benchmark` free of any learning import.
- **1.3 rejection, remediated RED-first**: `_body_digest` stripped every leading comment line, which in C and C++ swallows `#include` and `#define`, and in any language swallows an ordinary leading comment. Two files differing only in a buffer size on a `#define` hashed equal and were marked `identical_twin`, removing a real pair from every denominator. Measured before the fix: twelve file sides across six vendored pairs had directives inside the stripped prefix, including the Juliet stack-overflow pair whose whole point is that constant. The strip is now anchored to the grammar `harvest.provenance_header` writes: a leading comment counts as header only when it carries one of the eleven provenance keys followed by a colon, and the scan stops at the first line of code. Four regression tests pin both directions plus a corpus-wide assertion that no vendored excerpt loses a directive. The five twins, ten duplicate groups and nineteen straddling repositories are unchanged, so the committed hygiene artifact still holds.
- Suggestions applied: a mechanism claimed by two families now fails loud by name; the obligations tool's path-escape refusal and the endpoint's degrade-to-None on a provider error are now asserted; the TypeScript parse assertion is exact instead of a 25% allowance, which surfaced one real corpus defect (`lensops-kubernetesmanagement-fe4910` was harvested as a truncated destructuring parameter list and is not valid TypeScript on its own; it belongs to the 4.1 re-harvest).
- **Two findings escalated to the maintainer, both blocking task 3.8 rather than group 1.** (a) `untrusted_destination` contradicts the approved requirement sentence, which says "server-side request forgery": the family id is the key every published number carries, so `requirements.md` Req 1.1 and the brief's Decision 1 must be amended to the widened name, or the family renamed back and CWE-601's 34 rows re-homed. (b) The shipped taxonomy is flat, so `parent_child` is unreachable in production and the "partial credit for a parent-child relation" in Req 2.3 and 3.1 is inert at version 1; either declare real parents or record the inertness beside every published agreement number.
- Carry-forward: the nineteen straddling repositories cross the split by declaration, so `split_by_repository` cannot fix them and task 3.1's teacher rule inherits the leak until a maintainer re-declares those splits. The `preprocess_repository` cost of the three list tools should be measured in 2.5 before round zero.
- Task 2.1 (2026-09-06, RED first: 8 failing tests in `tests/test_learning_classify.py`; then green): `learning/classify.py` with three tiers, multi-label answers in taxonomy order, and `validate_family_labels` owning the taxonomy check the corpus loader deliberately does not do.
- Task 2.2 (2026-09-06, RED first: 5 failing tests in `tests/test_learning_classifier_report.py`; then green): `measure_classifier` and `write_review_queue`. Bug caught by the RED test: the classifier read the declared label it was being scored against, making agreement trivially 1.0; `classify_pair(..., use_declared=False)` is what the measurement uses. Measured on the 277 vendored pairs: tier one places 275 without a model call, 2 land in `unknown`, and the 176 memory-safety rows resolve to one family. `labeled` is 0 because no row carries a maintainer family label yet, and the report says so instead of inventing a score.
- Task 2.3 (2026-09-06, RED first: 8 failing tests in `tests/test_learning_scoring.py`; then green): `learning/scoring.py``score_pair_family`. Detection is a family-tagged finding inside the labeled function; text is never read. Findings of other families or outside the function are counted separately and never called leaks; a fabricated family id is counted as noise. K runs, majority outcome, disagreeing runs recorded as flips.
- Task 2.4 (2026-09-06, RED first: 7 failing tests in `tests/test_learning_aggregation.py`; then green): `aggregate` and `sign_test`. Recall at a fixed false-positive ceiling is forfeited entirely when the leak rate exceeds it; the directional-bias index separates "flags everything" from "silent on everything", which Youden cannot; change against a baseline is a two-sided exact sign test over per-pair flips, not a difference of means. `PairFamilyScore` gained a `slice` field (additive, filled from the case) so metrics can be reported per slice.
- Task 2.5 (2026-09-06, RED first: 9 failing tests in `tests/test_learning_detectors.py`; then green): `learning/detectors.py` with `FamilyConfig`, `Region`, `load_family_configs`, `run_family_detector`, `run_region_detectors`, `write_default_configs`. Deviations: the parts live in one `family.toml` plus the markdown and JSON-lines files, because two TOML files with three keys between them was more surface than value and `version` had no home in the design's list; configurations live under the project's learning directory, not inside the package, because a round writes them. `CURATED_TOOLS` is derived from `HUNTER_TOOLS` so the closed set cannot drift.
- Task 2.6 (2026-09-06, RED first: 6 failing tests in `tests/test_learning_verifiers.py`; then green): the optional oracle callback on `verdict_from_result`, inserted after the inconclusive guards and before the exit-code branch so a call without an oracle is the old decision tree, plus `verifier_for`, `StaticVerifier`, `NoVerifier` and `apply_verification`.
- Task 2.7 (2026-09-06, RED first: 8 failing tests in `tests/test_learning_canaries.py`; then green): `learning/canaries.py` with per-family templates, `CanaryVerifier` and `confirm_proven`. Deviation: templates live in `learning/canaries.py` rather than the design's `learning/verifiers/` directory; the property that matters, being outside every proposer write root, holds and is tested.
- Task 2.8 (2026-09-06, RED first: 6 failing tests in `tests/test_learning_journal.py`; then green): `learning/journal.py` with `RoundRecord`, `Attribution`, `LearningJournal`, `Archive` and `summarize`.
- Review round 2 (2026-09-06) APPROVED 2.1, 2.2, 2.4, 2.5, 2.8; REJECTED 2.3, 2.6, 2.7. The reviewer ran every check in a detached worktree at the group tip, re-derived the RED phase per task against each task's parent commit (8/5/8/7/9/6/8/6 failures, the four recorded claims exact), killed 15 mutants against the new tests, and hand-verified `sign_test` against the closed form for every (better, worse) with n <= 20.
- **2.3 rejection, remediated RED-first**: two of Req 4.1's five unscorable reasons had no code path, so a row whose language could not be parsed was reported as `unresolved_label`, a corpus labelling defect, when the cause was a tooling gap. The design's `unscorable_reason(case, *, parse_ok, ranges)` is reinstated as a public function, `score_pair_family` takes `parse_ok` and `entry_points`, and `missing_context` names a handler that arrived without the registration that makes it reachable, for the families whose bug shape is a missing check.
- **2.6 rejection, remediated RED-first**: `apply_verification` wrote `evidence_level="proven"`, which is not a member of the project's `EvidenceLevel`, so `verify_finding`, `is_report_verified` and `verify_judge` all raise `ValueError` on it. Requirements put the evidence ladder out of scope and the design's data contract specifies only the `verifier:<rung>` tag. Nothing in production called it yet, so the suite stayed green while task 3.4's wiring would have broken a scan. `apply_verification` now writes only the tag; `evidence_level_for` and `raise_to` are the single explicit bridge onto the ladder, and a test round-trips a proven claim through the project's own helpers.
- **2.7 rejection, remediated RED-first**: the injection canary opened its own in-memory database, inserted the token there and called the handler without ever giving it that connection, so the oracle could never fire; the fake sandbox runner returned the token regardless of the snippet, which is why no test noticed. It is now an execution canary whose marker only an interpreter could write, and each template is exercised end to end by running the rendered snippet against a vulnerable fixture and its fixed twin. That test immediately caught a second flaw: a fixed twin that echoes its input printed the oracle token and satisfied the oracle with no vulnerability at all, so an injected payload now carries a value unrelated to the token the oracle looks for. Recorded scope: the injection canary confirms command, code and template execution; a query-language injection whose payload never reaches an interpreter stays a suspicion.
- Suggestions applied: a configuration with an empty tool list is refused by name instead of producing a detector that silently reports nothing; the classifier abstains when its no-adapter fallback call also fails; the archive docstring states that the earliest winning version keeps its claim to a pair.
- Task-plan updates from the review: 2.2 and 2.4 claim the two criteria the maintainer added (2.6 and 3.7); 3.6 claims Req 6.2 with an observable that every non-target family directory is byte-identical across a round; 3.7 claims Req 3.5's K floor, 9.6's pair identities and 11.3's trajectories.
- **Escalated to the maintainer**: Req 6.1 lists "a skills set" among a detector configuration's parts, and neither the approved design nor the code has one, although the repository ships a skill router. Either Req 6.1 drops the skills set or a task wires it; the implementation followed the approved design, so this is a spec-internal conflict rather than a deviation.
- Carry-forward: the nineteen straddling repositories still cross the split by declaration and cap every holdout number these tasks produce; `unknown` serves as both the classifier's abstention and the generalist detector, which deserves a sentence in the design before a round ever targets it.
- Task 4.1 (2026-09-06, RED first: 21 of 23 failing tests in `tests/test_pair_registration_context.py`; then green, 26 tests). Measured before the change: **8 of the 10 vendored access-control rows had no named handler at all**, so every one of them would have scored `missing_context` forever rather than as a detector miss. After the re-harvest 9 of 10 are named and the tenth states its reason. What the repair needed, in order of how much it cost:
- **Class registration.** A flask-restx handler is a method; `@ns.route('/profile')` sits on the class. `extract_handler_context` now carries the enclosing class declaration and its decorators, so the excerpt is both reachable and parsable — a bare indented `def get` is neither.
- **Multi-line decorators and commented-out guards.** `@ns.doc(...)` wraps over four lines, and stopping at a continuation dropped the `@ns.route(...)` above it — the registration itself. `#@token_required` above a handler is not noise; in the two ThreatByte rows it is the entire bug, and the evidence field says so. `_decorator_start` now walks both.
- **Cross-file registration (design amendment, recorded in `design.md`).** connexion registers VAmPI's handlers by `operationId` in `openapi_specs/openapi3.yml`; Overleaf registers `exportProject` in `router.mjs`. No single-file excerpt can contain that, so a recipe may name `context = [...]`, the harvester writes the stanza per side under `<name>/context/<side>/<relpath>`, the catalog carries `[[pair.context]]`, and `_materialize_side` lays the documents out beside the excerpt at their repository-relative paths. `mapping._route_manifest_entry_points` reads `operationId` plus the operation's `security` block; `_js_statements` joins a registration that spans lines, because reading `webRouter.post(` alone finds neither the handler nor the middleware that guards it.
- **Anchors.** Real-Vuln-Benchmark trap rows take their fixed side from a different, correctly guarded handler of the same snapshot, so `handler_context` honours `line`/`fix_line` and a new `fix_function` gives the fixed side *its* registration instead of the vulnerable side's.
- **`innermost_function_bounds`** for brace languages, used only by this mode: `function_bounds_at` returns the outermost top-level group, which for a handler declared inside a React component is the whole 6000-line component. `hunk` and `enclosing` rows keep the bounds they were harvested with.
- **Fetch-then-write.** `harvest_recipe` fetches every blob before the first write, and the driver takes `--report` so a failed fetch leaves the reviewed excerpt exactly as it was and says why. A half-written re-harvest is indistinguishable from a real upstream change.
- Bugs the re-harvest exposed and fixed on the way: `.mjs` and `.cjs` were not in `LANGUAGE_BY_EXTENSION`, so the one pair vendored as an ES module had **no file target at all** — no entry point, no finding, a permanent miss; and `_python_route_access` matched auth words anywhere in a decorator, so `@ns.doc(description='... without proper authorization checks')` read the unguarded ThreatByte endpoint as `authenticated`, silencing every obligation on it. The guard is now the decorator's name, never its prose, and `@token_required` — the commonest Flask guard — is recognised at all.
- Corpus corrections: `lensops-kubernetesmanagement-fe4910` was labeled `setErrorMessage`, a React state setter, because the hunk landed in the wrong bounds; it is now `loadNamespaces` and the pair diff is exactly the fix. Its bug is a browser-side call omitting the header its siblings carry, not a missing guard on an inbound handler, so it carries `known_limit = "client_side_caller"` and leaves the denominators with a stated reason.
- Three existing tests encoded the old corpus and now encode the new one: the TypeScript parse pin (the lensops excerpt parses now), the classifier report (`labeled` 0 -> 4, the four seeded rows; the six `title`-tier agent rows carry a family but are not labels a maintainer stands behind), and the hygiene report ordering.
- **New honesty finding, recorded in the hygiene artifact as `cross_split_duplicates`: nine of the ten duplicate groups sit on both sides of the split.** Two forks of one project (`openniw-jobs-0fc947` train, `openniw-jobs-b74de3` holdout) and eight CVEs harvested twice under different names are byte-identical, so the train row teaches the holdout row exactly. `split_by_repository` cannot see it — the duplicates are different repositories. `duplicate_groups` is now ordered by name so the committed report does not reshuffle when one excerpt is re-harvested.
- Pre-existing drift the regeneration corrected, unrelated to this task: `catalog.toml` in git disagreed with a fresh `catalog_gen` run at the group-3 tip on two derived labels (`origin` -> `createApplication`, `normalizeDomain` -> `getGoogleProvider`). Verified in a detached worktree at `7a33756` before touching anything.
- Group 3 (2026-09-06). Tasks 3.1-3.9 implemented in `86ed54c`, `fcda1ee`, `e8e6baf`, `127dbef`, `ad7c98e`, `346cc70`, `16d9bc2`, `b955251`, `b8ad42c`, `7a33756`. RED counts, derived by the reviewer against each task's parent commit: 3.1 4/6, 3.2 6/6, 3.3 6/6, 3.4 3/5, 3.5 10/10, 3.6 12/12, 3.7 9/9, 3.8 7/7, 3.9 7/7. Deviations from the design, recorded here because they were not written down at the time: `learning.split` duck-types `PairCase` through `getattr` rather than importing it; `pairs.py` reaches `learning` through function-local imports rather than the design's injected callable; `run_baseline` takes a scan factory where the design named a client. One flaky test found and fixed in `346cc70` (`_is_vulnerable_side` keyed on `"-v" in str(root)`, which random temporary directory names matched about once in eight; harness noise recorded as detector noise would inflate every family's regression budget).
- Review round 3 (2026-09-06) APPROVED 3.4; REJECTED 3.1, 3.2, 3.3, 3.5, 3.6, 3.7, 3.8, 3.9. The reviewer ran every check in detached worktrees at the group tip against `71efc9c`, derived the RED phase per task, mutation-tested `decide`, the teacher rule, the snapshot restore and the round's revert paths (7 mutants on the acceptance rule, 6 killed and 1 equivalent), ran the three flakiness-prone files 61 times with varied `PYTHONHASHSEED` without a failure, and confirmed the three gate outputs byte-identical and the extra-free matrix unchanged. Every rejection was remediated RED-first; the numbers each defect was hiding are recorded below because they are the point of the spec.
- **3.2, the fixed side was judged against the vulnerable side's line ranges.** A fix that adds a guard above the handler moves its first line, and the reviewer measured that **191 of 227 pairs with a resolvable labeled span (84%) have different spans on the two sides**. A finding inside the fixed handler was therefore read as outside it: a leak recorded as silence, always in the direction that flatters Youden. `score_pair_family` now takes `fixed_ranges`; the fixed side's parse was already being computed and thrown away.
- **3.2 and 3.9, K runs was declared and never threaded.** `_evaluate_hunter_pair(..., runs=1)` had one call site that passed nothing, `evaluate_catalog` had no parameter at all, and `pairs --k-runs` was parsed and discarded; the only test grepped `--help`. K now flows `pairs --k-runs` -> `evaluate_catalog` -> `evaluate_pair` -> `_evaluate_hunter_pair`, defaults to three and refuses less, and the test counts scans per side.
- **3.2, the generalist could never be scored.** `pairs --hunter` fed the class-aware scorer findings with no `family:` tag, so it reported recall 0 by construction. `run_tool_hunter` now reads an optional `family` field into the tag (the design said it would) and `make_hunter_scan(taxonomy=...)` asks for one from the closed list. A generalist that names no family is unanswerable; one that names a wrong family is counted as noise, which is the honest treatment.
- **3.3 and 3.9, round zero measured every family with the generalist.** `run_baseline` took one `scan`, and the CLI handed it the `unknown` configuration; since `related(X, "unknown")` is `lateral` by the taxonomy's own decision that abstention earns no credit, **every family except `unknown` baselined at a structural zero** and every noise floor with it. `run_baseline` now takes a `scan_factory` and runs each family's own configuration, reports per slice as well as per family, and refuses fewer than three runs. This is the decision the group-2 review asked for on `unknown` serving as both abstention and generalist: it is the generalist for *scanning*, and it is never credited as a family for *scoring*.
- **3.5, the meta-agent proposer was a stub** that returned None after confirming the extra was present, so `ousast learning round` could never propose anything at any time. It now composes `MetaAgent.evolve` behind `harness_ext` against a scratch **copy** of the family directory — the live tree is never touched, because the round snapshots only after a proposal exists — and builds the proposal from the diff of that copy, so the one-file rule and the lever whitelist are enforced on what the agent did rather than on what it claimed. `learning round --proposals FILE` runs a recorded sequence offline, which is what makes the command exercisable without the extra at all.
- **3.6 and 3.1, the holdout refusal existed only as a vocabulary word.** `journal.py` documented `refused_holdout` and no code path could produce it; the covering test asserted `outcome == "refused_holdout" or outcome == "rejected"`, which passes whichever way the code behaves. The round now runs `refuse_if_holdout` over every corpus pair the proposal names — in its predictions and in the file it writes — and journals `refused_holdout` with the pair names. A memory entry that quotes a held-out pair looks exactly like a legitimate hard negative until the holdout number is read as if it meant something.
- **3.7, the sweep counted absolute failures, not negative flips.** Req 9.5 compares against the pre-change state; the code counted every pair of another family that was not `pair_correct` and compared that to a budget derived from the run-to-run flip rate. On the real corpus, where most pairs fail at round zero, that rejects **every** round for damage it did not do — a harness that then reports it cannot improve. Unscorable rows voted too, against Req 4.1. The sweep is now scored before and after the change, `_negative_flips` counts only pairs that were right and are not, unscorable rows leave the count, and attribution is derived from the same comparison instead of from absolute failures.
- **3.7, cost was metered once**, after the minibatch, the holdout and the whole cross-family sweep had all been paid for; Req 9.8 says stop, revert and record. The cap is now read after every stage. A stage that raises is journalled as `reverted`/`stage_failed` with its round directory before the exception leaves, because an exception that loses the record also loses the money the round already spent. A rejected round's version is archived: Req 9.7 exists so a configuration that lost overall but is the only one that ever won some pair is not forgotten.
- **3.8, the corrected lever number was published beside the earlier one, not in place of it** (Req 5.3), and the test named for that property never checked it. `publish` now rewrites the superseded claim to point at the regenerated section, idempotently, and the test asserts the old sentence is gone. Run against the real roadmap: the 2026-09-05 claim "human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353" now says it was measured with holdout pairs teaching and names the artifact that corrects it.
- **3.1, refusals were `continue` statements.** `Refusal.degradation()` existed and nothing called it. The mechanism lever now records every candidate it declined with the holdout pairs that taught it, leave-one-out records the pairs it kept out of the teaching set, and `improve` carries both into the round outcome. `semantic.seed` reported `split:train` for a train-split identical twin, which is simply untrue; an unscorable row now reports its own reason. `evaluate_profiles` and `evaluate_mechanism_profiles` are deliberately split-blind and now say so in their docstrings: they measure a ruleset and never learn from a pair, so a filter there would only shrink the evidence.
- Test-quality repairs the review asked for, each of which changed a number: two split tests were measuring an `x = 1` fixture that teaches nothing whatever the rule says, so both arms now use a pair that really carries a shape and the lever test asserts the same shape *is* admitted when a train pair taught it; the two CLI scoreboard tests asserted against output that was all zeros, and now use a side-aware detector stub (the shipped `scripted` stub answers twice, then goes quiet, and puts its finding outside the labeled function — three separate ways to make a family number zero); `ScriptedChatClient` gained an opt-in `cycle` so a stub cannot silently run out of answers mid-run; the sandbox-candidate assertion in `tests/test_learning_scan.py` compared verdict paths against a prefix no implementation would produce and now compares finding ids; the manifest's classifier block reported `measure_classifier([])`, all zeros, and now reports what this scan actually routed.
- Carry-forward, unchanged and still capping every holdout number: the nineteen straddling repositories, and — found during task 4.1 — **nine of the ten duplicate groups sit on both sides of the split**, which the repository-level split cannot see because the duplicates are different repositories.
- Task 4.2 (2026-09-06, RED first: 2 failing tests in `tests/test_fixture_difficulty.py`, and the check's own arm — `test_the_check_catches_a_fixture_that_is_easier_on_one_axis` — fails if the fixture hardening is reverted). Two parts:
- **The five twins declare their limit.** A content hash is evidence; the catalog row is the maintainer standing behind it, and without one a re-harvest that changes a byte turns a known-unscorable row back into a scored one with no review. A declared limit that names one of the vocabulary's own reasons keeps that spelling, so `identical_twin` declared and `identical_twin` computed stay one fact instead of splitting across two buckets in every aggregation. `semantic.seed._skip_reason` was reordered for the same reason: an unscorable row reports `unscorable:<reason>` before either the known-limit or the split branch can claim it. Consequence, recorded because it moves a published number: leave-one-out over vibe-py now runs on **30 rows, not 35** — a pair whose two sides are the same code cannot be held out against itself.
- **The fixture-difficulty check** (`learning/difficulty.py`), on three axes the corpus really varies on: binding hops from the identity source to the constrained use, registration distance (0 decorator on the handler, 1 statement elsewhere in the file, 2 another document), and guard distance (0 decorator, 1 call in the body, 2 nothing visible). The comparison is per axis over the whole set, because a suite covers a bug shape with several handlers between them. **Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).** Every shared handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no chain was longer than one binding — the checker was passing on a simpler world than the one it meets. `from_spec` was added to the fixture: the connexion shape those rows really have, a token validated by a call rather than a decorator and five bindings between the token and the constraint. The verdict is committed as `benchmarks/measurements/2026-09-06-fixture-difficulty.json` with both reaches and the history of the first measurement. One existing expectation moved with it (`from_spec` reads the user row and then the book: two real operations).
- Task 4.3 (2026-09-06, RED first: 10 failing tests in `tests/test_learning_cutoff.py`; then green, 11). The catalog generator fills `fix_date` from the recipe at the precision the recipe has: a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row therefore counts as post-cutoff only when its **earliest possible** date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the direction that would turn contamination into a result. `PairFamilyScore` carries the date, `aggregate(cutoff=...)` reports the same numbers over the post-cutoff rows beside the whole number and never instead of it, undated rows are counted on their own, and `publish` prints the slice with the cutoff it used. Without a configured cutoff every scoreboard says so rather than inventing one; `[learning].cutoff_date` has no default because guessing a model's training cutoff and publishing a contamination number from the guess would be worse than reporting none. Corpus dates as harvested: vibe-py all 2026, agent-vfc 67 of 69 in 2026, vfc spread 2014-2026, vfc-js 2014-2021.
- Task 5.1 (2026-09-06, RED first: 3 of 9 tests in `tests/test_degradation_matrix.py`; then green, 8 after one test was deleted rather than kept). The three gate outputs are captured at `71efc9c` — the commit before the first line of this feature — as `benchmarks/measurements/2026-09-06-gate-baseline.txt`, and a test reproduces them byte for byte on every run, in both matrices. Until now byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI, which is what makes it a gate rather than a habit. Six absences are each exercised once with the reason they record: HarnessX (`harnessx_unavailable`), the chat endpoint (`learning_endpoint_unavailable`, and the run still writes its artifact), the family configurations (`learning_configs_unavailable`), the network for pointer pairs (`pointer_pair_skipped`), the semantic extra (`language_unsupported` rather than a guess), and the sandbox (`sandbox_unavailable`, so a claim nothing could run is not confused with a claim that failed). One test written for this task was **deleted before the commit**: it asserted that this file contained a `without_<capability>` name per capability, which measures the file's own naming and nothing about the code.
- Review round 4 (2026-09-06) APPROVED 4.1, 4.2, 4.3; REJECTED the round-3 remediation on three counts. The reviewer re-derived the RED phase for group 4 (24 of 26 for 4.1, 3 for 4.2, 11 of 11 for 4.3 — **every count in the notes above was understated**, corrected here), killed 18 of 19 mutants, verified the 4.1 observable independently (2 of 10 access-control rows named at `7a33756`, 9 of 10 at `3e71e9e`, the tenth stating its reason), confirmed `_python_route_access` changed no access level anywhere in the corpus and `.mjs` changed exactly one snapshot, and reproduced the fixture-difficulty artifact number for number including its failing arm. Remediated RED-first:
- **The meta-agent proposer still could not propose.** `MetaAgent(inner_model=...)` takes a `ModelConfig`; it was handed a model name, a `# type: ignore` hid that from mypy, and a bare `except Exception` turned the resulting `AttributeError` into the degradation reason `meta_agent_failed` — which reads exactly like the model declining. Every other HarnessX composition site in the repository builds the config properly. `_meta_agent_class` and `_model_config` are now seams a test can pin without the extra, and a `TypeError`/`AttributeError`/`ImportError` from the call is re-raised: "we called the agent wrong" is not "the agent had nothing to propose".
- **`ousast pairs --hunter` crashed in its documented form.** The K floor landed with `--k-runs` still defaulting to 1, so the flag's absence raised `ValueError` from inside `evaluate_catalog`. The default is `None` (three), and a breach exits naming the flag. The covering test always passed `--k-runs 4`, which is why nothing noticed.
- **28 rows resolved a span on the vulnerable side and none on the fixed side** (12% of the corpus), so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said — the same flattering direction as the range defect, one-seventh the size. `fix_function` reached only the harvester; it now reaches the catalog and `PairCase`, `_spans` uses it for the fixed side, and the catalog generator derives it from the fixed excerpt itself for the 25 vibe-py trap rows that never declared one. A row that still resolves no fixed-side span reports `unresolved_label:fixed` instead of scoring silent. Measured after: **0 rows**.
- Suggestions applied: one spelling per corpus fact everywhere (`PairCase.unscorable` verbatim, no invented prefix); leave-one-out records non-teachers withheld for tier or an unscorable reason, not only holdout refusals; the hygiene test asserts the declared twins still hash equal, so a declaration that stopped being true would not be invisible; `is_post_cutoff` refuses a cutoff coarser than a day, which was the one over-claim path that existed, and month precision is pinned; `UNSCORABLE_REASONS` covers the whole vocabulary rather than `identical_twin` alone; the zero-run score carries its date; `_pairs_named_by` filters by substring before the regex.
- Carry-forward, measured 2026-09-06 after group 4 (`benchmarks/measurements/2026-09-06-trajectory-divergence.json`): **the K-run numbers this spec publishes are ensemble votes, not repeated measurements.** Across 120 runs of the injection family at temperature 0 with thinking disabled there was not one repeated tool-call trajectory, and outcome disagreement tracks trajectory divergence (30.9% at low similarity, 4.2% at mid). Consequences for anyone reading these numbers: Req 3.5's K is an ensemble over distinct search procedures; Req 8.2's noise floor measures how often an unconstrained search reaches the sink rather than the detector's uncertainty, so Req 9.5's regression budget grows with the search's own wandering; and the instability is entirely one-sided — every pair produced a single finding set on the fixed side, so specificity is solid and recall carries the caveat. `.kiro/specs/detector-optimizer/brief.md` proposes the repair, and it is a program change rather than a prompt change.
- Task 5.3 (2026-09-07, five rounds on `injection`, $2.17, `benchmarks/measurements/2026-09-07-evolve-rounds-injection.json`). The task was amended to target injection because access_control has one train teacher.
**Nothing the loop decided carried evidence, in either direction — five rounds, zero reliable decisions.** Round 2 rejected a change because **one pair of eight** moved, at p = 1.000 — the maximum the sign test can return. Round 3 accepted one because two of eight improved and none regressed, at p = 0.500, and that acceptance is now versioned as v1 and will be built on. On a family whose measured floor says **11 of 20 pairs disagree with themselves between runs of the unchanged detector**, both readings are what changing nothing looks like.
I overstated this once and correct it here: after rounds 1 and 2 I wrote that the rule was "close to structurally incapable of accepting anything". Round 3 accepted. The accurate statement is symmetric and worse — the rule accepts on noise as readily as it rejects on it.
- **Escalation to the maintainer, not applied here.** Req 9.5's threshold rule (reject if either side regresses, accept if one improves and neither regresses, tie rejects) should be replaced by an acceptance that survives the measured variance. `Archive` already holds the per-pair Pareto frontier and `acceptance.reliability` already computes the test; neither decides anything. The honest caveat is that at a minibatch of 8 and a holdout of 10 a reliability-gated rule would accept almost nothing either: **the corpus is the binding constraint, not the rule**, and no acceptance rule fixes a train split of twelve.
- **The v1 configuration rests on p = 0.500** and should probably be reverted before any further round builds on it. Left in place, and flagged, because reverting an accepted round is the maintainer's call.
- Round 1's rejection was correct but for my error, not the loop's insight: the `memory` lever writes `hard_negatives.jsonl`, which the detector renders as *"Known non-issues, do not report these"*, and I fed it the four vulnerability shapes the train split teaches. Round 5 re-ran the same content through `counterexamples` (*"Worked examples"*), which is the lever I meant — and was rejected because **one pair of eight** moved, at p = 1.000. **The experiment I designed to give the demo lever a fair test could not give it one.** The rule cannot distinguish a lever that helps from a lever that does nothing, so correcting my authoring error changed the reason for the verdict and not the verdict.
- What did work: the cross-family sweep caught a genuine negative flip in `access_control` in round 2 — a real regression found by the before/after comparison that replaced the absolute-failure count, which under the old rule would have rejected rounds for families that were already failing.
- **The finding that outranks the rounds (maintainer, 2026-09-07: "the error is not the model, the error is the elsewhere").** Eighteen defects were found in this session's measurement and round work. Every one was in this repository; none was the model's. The list is kept because it should change how the next bad number is triaged: the prose parser that discarded findings the model had already reached; the CLI that never read the cost meter it published; the baseline that lost everything on an exception, and again on a timeout; two keys that collapsed a slice into another, in two different modules; the pattern ruleset at a 12.1% ceiling; a harvester that truncated an excerpt at both ends; a family shape that could not see a permissive default inside a call argument; a lever wired to the file that says *do not report these*; a provider seam hard-coded to a vendor with no key; a proposer reason the journal threw away; a retry that did not cover a body which stopped mid-read; an exception path wrapped around half a round; log-probabilities read from the wrong object; `.mjs` absent from the language table; and an access-level check that matched prose inside a decorator. Three of them were the same shape — fixing an instance and missing the class. **The default assumption for an unexplained number is that we broke it.**`deepseek-v4-pro` scoring 46.7% against flash's 83.3% is unexplained and is therefore, by this prior, ours until shown otherwise.
