Open_Ultra_SAST/.kiro/specs/learning-harness/tasks.md

Implementation Plan

• 1. Foundation: taxonomy, endpoint, corpus fields, grammar

• 1.1 Closed family taxonomy with a verifier per family

  • Ten families exactly as decided, each with its CWE set, mechanism set, verifier kind (canary, static, none) and description; unknown always present with no verifier; a version string.
  • Loader rejects an unknown field, an eleventh family, a missing verifier kind or a CWE claimed by two families, naming the offender; relation lookup answers same, parent–child, lateral or fabricated for any two ids.
  • Observable: the taxonomy loads from the ruleset directory, a fixture with a fabricated family fails by name, and every mechanism id in the mechanism vocabulary resolves to exactly one family.
  • Requirements: 1.1, 1.2, 1.3, 1.4

• 1.2 Independent chat endpoint with the DeepSeek adapter

  • Endpoint resolution order: explicit override, scripted client modes, DeepSeek key and base URL, then OpenRouter; the embedding client keeps reading only its own variables; [models] gains the hunter and judge defaults and the endpoint keys, [learning] gains its section.
  • Adapter disables thinking and sends temperature zero for detector and classifier calls, keeps the provider's reasoning field on assistant messages so a caller can replay it, supports JSON-object mode with one retry on empty content and optional log-probabilities, and reports normalized usage and cost from the provider's cache-hit fields.
  • Observable: with both provider keys set the hunter client targets the DeepSeek base URL without a version suffix while the embedding client still targets OpenRouter; a scripted client is chosen under the existing environment flag; a recorded request shows thinking disabled and the reasoning field retained on the reply.
  • Requirements: 6.6, 11.4

• 1.3 Additive corpus fields and hygiene at load

  • Expected rows accept an opaque family id (validated later by the classifier task, so the corpus module never imports the learning package); a pair carries a computed unscorable reason (identical twin by normalized body comparison with headers stripped, computed only when the catalog does not already declare a known limit) and is never dropped; duplicates across pairs are reported before any split; a repository-and-date split helper assigns train or holdout to rows that lack a split, keeps declared splits, and reports rows from one repository that straddle the split; pair signals take a split filter.
  • Observable: the five identical vibe-py twins load with the identical-twin reason and the existing catalogs load with unchanged row counts; the straddle report on the current catalogs is produced and committed.
  • Requirements: 4.1, 4.2, 4.3, 4.4

• 1.4 TypeScript rows parse under the extra

  • The semantic extra gains the TypeScript grammar and the parser maps TypeScript and TSX to it; the extra-free suite still skips them with the existing marker.
  • Observable: every vendored TypeScript pair parses on both sides under the extra, and the parser reports no unsupported-language result for TypeScript or TSX.
  • Requirements: 10.1

• 1.5 Curated hunter tools and hunter-loop parameters

  • Four new repository tools beside the existing three: resolve a definition by symbol, list entry points, list flows for a function, list obligations for a function, each clamped to the scanned repository and returning identifiers and line spans only; the hunter loop gains prompt, context files, tool set, character limit and tag parameters with today's behavior as defaults and replays the provider's reasoning field across tool turns.
  • Observable: the existing hunter tests pass unchanged with default parameters; each new tool answers on the shared handler fixture and refuses a path outside the repository; a recorded tool-turn conversation shows the reasoning field replayed.
  • Requirements: 6.4, 6.5

• 2. Core: classifier, scorer, detectors, verifiers, journal

• 2.1 (P) Classifier tiers

  • Tier one decides from rule id, sink family, obligation kind, mechanism id or labeled CWE without a model; tier two asks the chat endpoint in JSON-object mode with thinking disabled and averages log-probabilities, falling to unknown under a threshold; tier three is unknown; every answer records its tier; results are multi-label; loaded catalog family labels are validated against the taxonomy here and a fabricated family is rejected by name.
  • Observable: on the shared handler fixture tier one classifies the leaky handler as access control without any client; with a scripted client an off-vocabulary answer becomes unknown; a catalog with a fabricated family is rejected by name.
  • Requirements: 2.1, 2.2
  • Boundary: Classifier

• 2.2 Classifier measurement and review queue

  • Report: agreement with maintainer labels on reviewed pairs with parent–child partial credit and no credit for lateral or fabricated ids, confusion matrix per family, accuracy against a random router and an oracle router, counts per tier and unknown; disagreements append to a review queue and never change a label; the whole corpus, including free-text and other rows, is classified with per-family counts.
  • Observable: the report on the vendored corpus lists the vfc other rows by resulting family and the queue holds every disagreement once with both answers and the tier.
  • Observable: the report also states whether hierarchical partial credit could apply.
  • Requirements: 2.3, 2.4, 2.5, 2.6

• 2.3 (P) Per-pair family scoring

  • Detection only for a finding whose family tag relates same or parent–child to the labeled family and whose line lies in the labeled function, with the maximum penalty for a fabricated family; silence only when no such finding lies in the labeled function on the fixed side; other findings counted separately, never as leaks; text is never inspected; outcomes pair-correct, both-flagged, both-silent, reversed and unscorable; the unscorable reason also covers unsupported language, unresolved label and missing context.
  • Observable: a synthetic run set where every fixed side carries an other-family finding scores full silence; a finding with a fabricated family never counts; a pair in an unsupported language scores unscorable with that reason.
  • Requirements: 3.1, 3.2, 3.6, 4.1
  • Boundary: Scorer

• 2.4 Aggregation, noise handling and denominators

  • Per family and slice: outcome counts, recall, silence, Youden, directional-bias index, a fixed-false-positive-rate view; K runs per pair with the majority outcome, negative flips per family against a baseline set, and a paired reliable-change test; unscorable rows listed by reason and excluded from every denominator; every metrics block carries the taxonomy version and the scorable and unscorable counts.
  • Observable: a single-run difference below the reliable-change threshold reports no change while a consistent flip across runs does; the unscorable count and taxonomy version appear in every metrics block.
  • Observable: every metrics block states whether hierarchical partial credit could apply.
  • Requirements: 3.3, 3.4, 3.5, 3.7, 4.5

• 2.5 (P) Per-family detector configurations and the family detector runner

  • A configuration directory per family: prompt under a stated length cap, checklist, tool list from the curated set only, step and cost budget, maximum characters, hard negatives and counterexamples, version; the loader rejects an over-cap prompt, an unknown tool or a missing family; this task ships the loader and a test fixture configuration, the shipped per-family directories are written by round zero.
  • The runner takes the families admitted by the classifier as input, builds the detector's user message from the whole file plus definitions its tools resolve, runs the hunter loop with the family's prompt, tools and budgets, and tags each finding with exactly one family tag and one detector-version tag; the generalist is the unknown family's configuration.
  • Observable: a family run on the shared fixture with a scripted client yields findings tagged with that family and version and uses only the family's tool list; a configuration whose prompt exceeds the cap fails to load by name.
  • Depends: 2.1
  • Requirements: 6.1, 6.2, 6.3, 6.4, 6.5
  • Boundary: Detectors

• 2.6 (P) Verifier registry and the oracle callback

  • The regress verdict accepts an optional oracle callback and keeps exit-code behavior otherwise; a registry keyed by the taxonomy's verifier kind returns a canary, static or no-op verifier; the static verifier raises access-control claims to static corroboration when the obligations checker reports the same function; the no-op verifier leaves suspicion.
  • Observable: with the fake sandbox runner a result whose output carries the token yields triggerable while a nonzero exit without the token does not; a family with no verifier keeps suspicion on the shared fixture; the access-control claim on the leaky handler reaches static corroboration.
  • Requirements: 7.1, 7.3, 7.4
  • Boundary: Verifiers, Regress

• 2.7 Canary templates and the second judge

  • Family snippet templates for injection, file path and deserialization plant a canary (row in an in-memory database, file under the scratch mount, marked object) and print the token only when the vulnerability fires; the second judge asks the judge model with the snippet, oracle output and claim and answers confirm or not in JSON-object mode; a claim is published as proven only when it confirms; templates, oracle code and gold labels live outside every proposer write root.
  • Observable: every canary template passes the existing snippet safety check; a scripted judge that declines keeps the claim below proven; templates and oracle code live under the verifier directory and gold labels under the corpus directory.
  • Requirements: 7.2, 7.5

• 2.8 (P) Round journal, archive and rejected buffer

  • Append-only round records with hypothesis, levers, predicted affected and at-risk families, outcome, reason, target deltas, per-family sweep flips, attribution (flipped predicted, flipped unpredicted, precision), cost, taxonomy and configuration versions; a per-family archive of which configuration version wins which pair; a rejected-hypothesis buffer per family; every persisted text passes redaction.
  • Observable: a record round-trips through the file; the buffer returns only the target family's rejected hypotheses; the archive names one winner per pair per family; a secret literal written into a record is stored redacted.
  • Requirements: 9.3, 9.6, 9.7, 11.3
  • Boundary: Journal

• 3. Integration: split, scoring paths, rounds, proposer, publish, CLI

• 3.1 One teacher rule across every learning path

  • Teachers are vendored, gating-tier, train-split, scorable pairs; a refusal names any candidate taught by or touching a holdout pair; the exporter, leave-one-out, per-profile evaluation, mechanism-profile evaluation, mechanism admission and pair signals all obtain teachers through the rule and record refusals as degradations; the mechanism lever's holdout number is re-measured under the rule and its raw artifact committed for the publish task.
  • Observable: leave-one-out on the toy slice seeds from train pairs only; a candidate shape taught by a holdout pair is refused by name; the re-measured artifact is in the measurements directory.
  • Depends: 1.3
  • Requirements: 5.1, 5.2, 5.3

• 3.2 Family scoring on the pairs hunter path

  • The pair scorer's hunter path takes an injected family matcher and leak rule from the learning package, classifies each pair once, runs the admitted family detectors plus the generalist K times per side, and emits family outcomes and metrics; the overlay and inventory paths stay untouched.
  • Observable: the pairs command with a scripted client on the shared fixture reports a family outcome per pair; no text-token matching remains on the hunter path; the pair gate output is byte-identical to before.
  • Depends: 2.1, 2.4, 2.5, 2.6
  • Requirements: 3.1, 3.2, 3.3, 3.5, 11.5

• 3.3 Round zero and the noise floor

  • Clone the current hunter prompt into every family configuration; run each family K = 5 times over the scorable vendored pairs of the selected slices at deterministic settings; record the per-family negative-flip rate between runs as that family's regression budget; persist noise floors and round-zero artifacts keyed by detector model so another model can be baselined and compared without touching configurations; the web families over vibe-py and agent-vfc are the default selection and the memory-safety family over vfc is selectable but never gates.
  • Observable: with a scripted client round zero writes a noise-floor file with one budget per family and a round-zero report per slice; a second baseline under another model name writes a separate artifact and the comparison table lists both.
  • Depends: 3.2
  • Requirements: 8.1, 8.2, 8.3, 8.4

• 3.4 Family detectors, verifiers and tags in scans

  • In a standard scan each classified region runs its admitted family detectors plus the generalist from the directories round zero wrote (a missing family directory is a recorded degradation and that family's detector is skipped), claims pass through the family's verifier, findings carry family, detector and verifier tags in reports and SARIF, and the manifest gains a learning block with taxonomy version, per-family counts, the classifier report and the current round; the detection and map gates stay byte-identical.
  • Observable: a standard scan of the shared fixture with a scripted client writes findings tagged with a family, a detector version and a verifier rung, and the manifest block; the detection and map gate outputs diff empty.
  • Depends: 3.3, 2.6
  • Requirements: 6.3, 7.1, 11.5

• 3.5 Proposer protocol, scripted proposer and the HarnessX proposer

  • Failure facts for one family: misses and leaks with pair, function, sink or check, line, verifier outcome and run outcomes, plus the rejected buffer and the current configuration; a proposal is one hypothesis, one lever (tool, checklist, counterexample, memory, prompt), one file-level change inside the family directory, and predicted affected and at-risk families; the proposer never receives holdout inputs, outputs or scores.
  • A scripted proposer for tests and offline rounds; a HarnessX proposer composing the meta-agent with write roots limited to the family directory, degrading to a recorded reason without the extra.
  • Observable: the scripted proposer returns a proposal from a facts fixture; the HarnessX proposer is skipped with the reason recorded when the extra is absent and, when present, writes only inside the family directory; the write-root list excludes the verifier and corpus directories.
  • Depends: 2.5, 2.8
  • Requirements: 9.1, 9.2, 5.4, 7.6

• 3.6 Acceptance rule and directory revert

  • Acceptance requires train and holdout non-negative with one strictly positive and every other family within its budget; ties reject; a family directory snapshot restores byte for byte on rejection or overrun; refusals for holdout contact, length cap and lever set are outcomes with reasons.
  • Observable: a helpful change is accepted, a tie is rejected, a change that breaches another family's budget is rejected with the family named, and a revert restores the directory byte for byte.
  • Observable: every family other than the target has a byte-identical directory before and after the round, whatever the outcome.
  • Depends: 3.3
  • Requirements: 6.2, 9.5

• 3.7 Evolve round orchestration, cost and journal

  • One round: facts, proposal, refusals, train minibatch, target-family holdout, sweep over every other family's holdout, K-run scoring at every stage as the scorer requires, accept (freeze, bump version, archive) or revert; cost metered per stage with revert and a recorded reason on overrun; a journal record and the rejected buffer written for every outcome; a round directory holds the snapshot, proposal, scores and trajectories and a repeated round number is refused.
  • Observable: with the scripted proposer each of accepted, rejected, budget-breach and cost-overrun leaves one journal record with attribution and a round directory; a repeated round number is refused.
  • The round directory holds the pair identities behind the attribution counts and every detector trajectory, redacted, so a round can be replayed and its attribution audited; every stage scores at least three runs per pair.
  • Depends: 3.5, 3.6
  • Requirements: 3.5, 9.4, 9.6, 9.7, 9.8, 9.9, 11.3

• 3.8 Publish from artifacts

  • One entry point regenerates every family and slice number from committed artifacts, writes a markdown table with scorable and unscorable counts, taxonomy version, noise floor and cost per point beside every number, replaces the roadmap's measured section between markers, and writes the raw artifacts under the measurements directory.
  • Observable: running the publish entry point twice without new artifacts produces byte-identical output; the corrected mechanism-lever number from the re-measurement artifact replaces the earlier roadmap value.
  • Depends: 3.1, 3.3
  • Requirements: 11.1, 11.2, 4.5, 5.3

• 3.9 Learning subcommands

  • Subcommands for classify, score, baseline, round and publish with the flags the design names; the pairs hunter path gains the K-runs flag; every step runs under the harness runtime with degradations for missing extras, endpoints or sandbox.
  • Observable: each subcommand exits zero on the shared fixture with a scripted client and writes its artifact; the roadmap section written by the publish subcommand carries the command line that produced it; a missing endpoint records the reason and the remaining steps complete.
  • Depends: 3.7, 3.8
  • Requirements: 11.1, 11.4

• 4. Corpus repairs

• 4.1 Re-harvest absence rows with registration context

  • Absence rows in vibe-py and agent-vfc gain their family and the handler-context mode in the recipes; the maintainer re-harvests them over the network; every re-harvested excerpt keeps its license line, passes redaction and keeps the decorators and registration statements; failed fetches keep the old excerpt with a recorded reason; catalogs regenerate idempotently.
  • Observable: the vampi and threatbyte excerpts contain their registration lines, the entry-point mapper names their handlers, and the catalogs load with unchanged vendored counts.
  • Depends: 1.3
  • Requirements: 10.2

• 4.2 Identical twins and the fixture-difficulty check

  • The five identical vibe-py twins carry the identical-twin known-limit reason in the catalog; a test derives a difficulty vector for a unit fixture (binding hops to the identity source, registration present, cross-file registration) and for the corpus rows its docstring names, and fails when the fixture is strictly easier.
  • Observable: the twins are excluded from every denominator with their reason listed; the shared obligations fixture is compared against the re-harvested vampi rows and the check's verdict is recorded.
  • Depends: 4.1
  • Requirements: 10.3, 10.4

• 4.3 Fix dates and the post-cutoff slice

  • Catalog rows gain a fix date populated by the catalog generator from recipe metadata where available; rows without a date are counted separately; rows dated after the configured detector cutoff form a separately reported slice in every scoreboard.
  • Observable: the scoreboard shows the post-cutoff slice with its own counts, the cutoff date it used and the undated count.
  • Depends: 1.3, 2.4
  • Requirements: 10.5

• 5. Validation

• 5.1 Gates, matrices and degradation paths

  • Detection, map and local pair gate outputs byte-identical to the pre-feature commit; the extra-free suite green with semantic and HarnessX tests skipping; each optional dependency's absence exercised once with its recorded reason.
  • Observable: the three gate outputs diff empty against the recorded baseline and both test matrices pass.
  • Requirements: 11.4, 11.5

• 5.2 Live round zero

  • Round zero through the DeepSeek endpoint on the vibe-py and agent-vfc web families with K = 5, plus the memory-safety family over the vfc slice as a measured, non-gating run; artifacts, noise floors and cost per pair committed; the classifier report and review queue produced on the same run; a second baseline with the judge model on one slice for the model comparison table.
  • Observable: the published table shows per-family outcomes, noise floors and cost for both models and for the vfc memory-safety family, and the roadmap section is regenerated from it.
  • Depends: 3.9, 4.1
  • Requirements: 8.3, 8.4, 11.1, 11.2

• 5.3 First evolve rounds

  • Three rounds with the scripted proposer on the access-control family using the obligation shapes as the first memory entries, then one round with the HarnessX proposer; every round journaled with attribution; the roadmap regenerated.
  • Observable: the journal holds four records with outcomes and attribution precision, at least one accepted change is versioned in its family directory, and no other family exceeded its budget in any accepted round.
  • Depends: 5.2
  • Requirements: 9.9

Task-plan review notes

• Sanity review round 1 (2026-09-06): NEEDS_FIXES, eleven findings, all repaired (K-run policy honors Req 3.5; per-model baselines added to the design; family validation moved out of the corpus loader; reasoning replay owned by the hunter loop; Req 5.3 split into re-measure and publish; scoring path split into pairs and scan tasks; oversized tasks split; fix-date prerequisite added; fixture-difficulty check depends on the re-harvest; memory-safety family measured on vfc; detector task depends on the classifier and round zero owns the shipped directories; observables re-anchored).
• Sanity review round 2 (2026-09-06): NEEDS_FIXES with three local items, repaired without a third review under the bounded loop: task 1.5 delivers the four curated tools and the hunter-loop parameters that 2.5 relied on; the scan-path task now follows round zero and degrades when a family directory is missing; the write-root observable moved to the HarnessX proposer task with 7.6. Non-blocking notes applied: the split helper assigns as well as reports (4.4), 3.2 depends on 2.1 rather than 2.2.

Implementation Notes

• Requirements approved 2026-09-06, design approved 2026-09-06, tasks approved 2026-09-06 (maintainer, in session).
• Task 1.1 (2026-09-06, RED first: 9 failing tests in tests/test_learning_families.py, ModuleNotFoundError: openultrasast.learning; then green): learning/families.py and ruleset/families.toml with the ten decided families, each carrying its CWE set, mechanism set, verifier kind and description. Decisions taken while writing the data: (a) unknown carries the six mechanisms that span families (source_reaches_sink, container_taint, validation_strength, redos, cross_artifact, other) and the two weaknesses we deliberately do not target (CWE-20 generic validation, CWE-835 infinite loop), so a row we considered and declined is visible instead of unresolved; an earlier draft said unknown had none. (b) The maintainer's "server-side request forgery" family is named untrusted_destination and also covers open redirect (CWE-601), because both are untrusted input choosing a request destination and the corpus carries 34 open-redirect rows that would otherwise be unclassifiable. (c) The taxonomy is flat: parent is an optional field no shipped family uses, exercised by a fixture, because Req 2.3 and 3.1 need a parent-child arm and inventing a hierarchy would be worse than leaving it empty. (d) Abstention earns no partial credit: related("unknown", X) is lateral. All 33 labeled corpus weaknesses resolve, none is claimed twice, and every mechanism id belongs to exactly one family.
• Task 1.2 (2026-09-06, RED first: 8 failing tests in tests/test_learning_endpoint.py; then green): learning/endpoint.py resolves override, then the scripted flag, then a configured endpoint, then DeepSeek, then OpenRouter; the embedding client is untouched. provider/openrouter.py gains complete_chat_raw (whole payload, for usage) and an extra_body passthrough; ChatResponse gains reasoning and mean_logprob. Design deviation: ChatEndpoint deliberately holds no api_key (the design listed one) so the record is safe to journal; the key stays on the client. [models] gains judge, chat_base_url, chat_api_key_env; [learning] is new. ModelConfig.hunter keeps its None default on purpose: defaulting it would silently enable LLM stages in existing scans, so the DeepSeek model defaults live in resolve_models.
• Task 1.3 (2026-09-06, RED first: 7 failing tests in tests/test_pair_hygiene.py; then green): ExpectedFinding.family (opaque; the classifier validates it), PairCase.unscorable, PairCase.split_declared, PairCase.fix_date (the generator fills it in 4.3), duplicate_groups, split_by_repository with SplitReport, and a split filter on build_pair_signals. Measured and committed as benchmarks/measurements/2026-09-06-corpus-hygiene.json: 19 repositories straddle the train/holdout split (a learning path can be taught by one file of a repository and evaluated on another) and 10 groups of pairs share a vulnerable excerpt byte for byte, mostly the same fix harvested twice at different granularity. Only three vendored rows (the local slice) declare no split.
• Task 1.4 (2026-09-06, RED first: 2 failing tests in tests/test_typescript_grammar.py; then green): the semantic extra gains tree-sitter-typescript; grammar_for maps a language to (module, attribute) because the wheel ships two grammars, and parse_with_cst picks the JSX-aware one for a .tsx path while the parsed file keeps the language it was given. Eighteen agent-vfc rows plus the TypeScript absence rows now parse on both sides.
• Task 1.5 (2026-09-06, RED first: 5 of 6 tests in tests/test_hunter_curated_tools.py failing; then green): hunter_tools gains read_definition, entry_points, flows, obligations; run_tool_hunter gains system_prompt, user_prompt, context_files, tools, max_chars and tags, all defaulting to today's behaviour, and replays the provider's reasoning field on the next turn. Deviations: read_definition returns the definition text as well as its span, and includes preceding decorator lines, because a resolver that returns only line numbers forces another read_file and a handler's decorators are exactly what the obligations work showed to matter; the three list tools return identifiers and spans only. No caching: a stale view of a tree a round may have rewritten is worse than a reparse.
• Review round 1 (2026-09-06) APPROVED 1.1, 1.2, 1.4, 1.5; REJECTED 1.3. The reviewer ran every check in a detached worktree at the group-1 tip and confirmed RED-first by running each new test file against the pre-feature commit (37 failures there, all passing after), the three gate outputs byte-identical, and pairs/benchmark free of any learning import.
  • 1.3 rejection, remediated RED-first: _body_digest stripped every leading comment line, which in C and C++ swallows #include and #define, and in any language swallows an ordinary leading comment. Two files differing only in a buffer size on a #define hashed equal and were marked identical_twin, removing a real pair from every denominator. Measured before the fix: twelve file sides across six vendored pairs had directives inside the stripped prefix, including the Juliet stack-overflow pair whose whole point is that constant. The strip is now anchored to the grammar harvest.provenance_header writes: a leading comment counts as header only when it carries one of the eleven provenance keys followed by a colon, and the scan stops at the first line of code. Four regression tests pin both directions plus a corpus-wide assertion that no vendored excerpt loses a directive. The five twins, ten duplicate groups and nineteen straddling repositories are unchanged, so the committed hygiene artifact still holds.
  • Suggestions applied: a mechanism claimed by two families now fails loud by name; the obligations tool's path-escape refusal and the endpoint's degrade-to-None on a provider error are now asserted; the TypeScript parse assertion is exact instead of a 25% allowance, which surfaced one real corpus defect (lensops-kubernetesmanagement-fe4910 was harvested as a truncated destructuring parameter list and is not valid TypeScript on its own; it belongs to the 4.1 re-harvest).
  • Two findings escalated to the maintainer, both blocking task 3.8 rather than group 1. (a) untrusted_destination contradicts the approved requirement sentence, which says "server-side request forgery": the family id is the key every published number carries, so requirements.md Req 1.1 and the brief's Decision 1 must be amended to the widened name, or the family renamed back and CWE-601's 34 rows re-homed. (b) The shipped taxonomy is flat, so parent_child is unreachable in production and the "partial credit for a parent-child relation" in Req 2.3 and 3.1 is inert at version 1; either declare real parents or record the inertness beside every published agreement number.
  • Carry-forward: the nineteen straddling repositories cross the split by declaration, so split_by_repository cannot fix them and task 3.1's teacher rule inherits the leak until a maintainer re-declares those splits. The preprocess_repository cost of the three list tools should be measured in 2.5 before round zero.
• Task 2.1 (2026-09-06, RED first: 8 failing tests in tests/test_learning_classify.py; then green): learning/classify.py with three tiers, multi-label answers in taxonomy order, and validate_family_labels owning the taxonomy check the corpus loader deliberately does not do.
• Task 2.2 (2026-09-06, RED first: 5 failing tests in tests/test_learning_classifier_report.py; then green): measure_classifier and write_review_queue. Bug caught by the RED test: the classifier read the declared label it was being scored against, making agreement trivially 1.0; classify_pair(..., use_declared=False) is what the measurement uses. Measured on the 277 vendored pairs: tier one places 275 without a model call, 2 land in unknown, and the 176 memory-safety rows resolve to one family. labeled is 0 because no row carries a maintainer family label yet, and the report says so instead of inventing a score.
• Task 2.3 (2026-09-06, RED first: 8 failing tests in tests/test_learning_scoring.py; then green): learning/scoring.py score_pair_family. Detection is a family-tagged finding inside the labeled function; text is never read. Findings of other families or outside the function are counted separately and never called leaks; a fabricated family id is counted as noise. K runs, majority outcome, disagreeing runs recorded as flips.
• Task 2.4 (2026-09-06, RED first: 7 failing tests in tests/test_learning_aggregation.py; then green): aggregate and sign_test. Recall at a fixed false-positive ceiling is forfeited entirely when the leak rate exceeds it; the directional-bias index separates "flags everything" from "silent on everything", which Youden cannot; change against a baseline is a two-sided exact sign test over per-pair flips, not a difference of means. PairFamilyScore gained a slice field (additive, filled from the case) so metrics can be reported per slice.
• Task 2.5 (2026-09-06, RED first: 9 failing tests in tests/test_learning_detectors.py; then green): learning/detectors.py with FamilyConfig, Region, load_family_configs, run_family_detector, run_region_detectors, write_default_configs. Deviations: the parts live in one family.toml plus the markdown and JSON-lines files, because two TOML files with three keys between them was more surface than value and version had no home in the design's list; configurations live under the project's learning directory, not inside the package, because a round writes them. CURATED_TOOLS is derived from HUNTER_TOOLS so the closed set cannot drift.
• Task 2.6 (2026-09-06, RED first: 6 failing tests in tests/test_learning_verifiers.py; then green): the optional oracle callback on verdict_from_result, inserted after the inconclusive guards and before the exit-code branch so a call without an oracle is the old decision tree, plus verifier_for, StaticVerifier, NoVerifier and apply_verification.
• Task 2.7 (2026-09-06, RED first: 8 failing tests in tests/test_learning_canaries.py; then green): learning/canaries.py with per-family templates, CanaryVerifier and confirm_proven. Deviation: templates live in learning/canaries.py rather than the design's learning/verifiers/ directory; the property that matters, being outside every proposer write root, holds and is tested.
• Task 2.8 (2026-09-06, RED first: 6 failing tests in tests/test_learning_journal.py; then green): learning/journal.py with RoundRecord, Attribution, LearningJournal, Archive and summarize.
• Review round 2 (2026-09-06) APPROVED 2.1, 2.2, 2.4, 2.5, 2.8; REJECTED 2.3, 2.6, 2.7. The reviewer ran every check in a detached worktree at the group tip, re-derived the RED phase per task against each task's parent commit (8/5/8/7/9/6/8/6 failures, the four recorded claims exact), killed 15 mutants against the new tests, and hand-verified sign_test against the closed form for every (better, worse) with n <= 20.
  • 2.3 rejection, remediated RED-first: two of Req 4.1's five unscorable reasons had no code path, so a row whose language could not be parsed was reported as unresolved_label, a corpus labelling defect, when the cause was a tooling gap. The design's unscorable_reason(case, *, parse_ok, ranges) is reinstated as a public function, score_pair_family takes parse_ok and entry_points, and missing_context names a handler that arrived without the registration that makes it reachable, for the families whose bug shape is a missing check.
  • 2.6 rejection, remediated RED-first: apply_verification wrote evidence_level="proven", which is not a member of the project's EvidenceLevel, so verify_finding, is_report_verified and verify_judge all raise ValueError on it. Requirements put the evidence ladder out of scope and the design's data contract specifies only the verifier:<rung> tag. Nothing in production called it yet, so the suite stayed green while task 3.4's wiring would have broken a scan. apply_verification now writes only the tag; evidence_level_for and raise_to are the single explicit bridge onto the ladder, and a test round-trips a proven claim through the project's own helpers.
  • 2.7 rejection, remediated RED-first: the injection canary opened its own in-memory database, inserted the token there and called the handler without ever giving it that connection, so the oracle could never fire; the fake sandbox runner returned the token regardless of the snippet, which is why no test noticed. It is now an execution canary whose marker only an interpreter could write, and each template is exercised end to end by running the rendered snippet against a vulnerable fixture and its fixed twin. That test immediately caught a second flaw: a fixed twin that echoes its input printed the oracle token and satisfied the oracle with no vulnerability at all, so an injected payload now carries a value unrelated to the token the oracle looks for. Recorded scope: the injection canary confirms command, code and template execution; a query-language injection whose payload never reaches an interpreter stays a suspicion.
  • Suggestions applied: a configuration with an empty tool list is refused by name instead of producing a detector that silently reports nothing; the classifier abstains when its no-adapter fallback call also fails; the archive docstring states that the earliest winning version keeps its claim to a pair.
  • Task-plan updates from the review: 2.2 and 2.4 claim the two criteria the maintainer added (2.6 and 3.7); 3.6 claims Req 6.2 with an observable that every non-target family directory is byte-identical across a round; 3.7 claims Req 3.5's K floor, 9.6's pair identities and 11.3's trajectories.
  • Escalated to the maintainer: Req 6.1 lists "a skills set" among a detector configuration's parts, and neither the approved design nor the code has one, although the repository ships a skill router. Either Req 6.1 drops the skills set or a task wires it; the implementation followed the approved design, so this is a spec-internal conflict rather than a deviation.
  • Carry-forward: the nineteen straddling repositories still cross the split by declaration and cap every holdout number these tasks produce; unknown serves as both the classifier's abstention and the generalist detector, which deserves a sentence in the design before a round ever targets it.
• Task 4.1 (2026-09-06, RED first: 21 of 23 failing tests in tests/test_pair_registration_context.py; then green, 26 tests). Measured before the change: 8 of the 10 vendored access-control rows had no named handler at all, so every one of them would have scored missing_context forever rather than as a detector miss. After the re-harvest 9 of 10 are named and the tenth states its reason. What the repair needed, in order of how much it cost:
  • Class registration. A flask-restx handler is a method; @ns.route('/profile') sits on the class. extract_handler_context now carries the enclosing class declaration and its decorators, so the excerpt is both reachable and parsable — a bare indented def get is neither.
  • Multi-line decorators and commented-out guards. @ns.doc(...) wraps over four lines, and stopping at a continuation dropped the @ns.route(...) above it — the registration itself. #@token_required above a handler is not noise; in the two ThreatByte rows it is the entire bug, and the evidence field says so. _decorator_start now walks both.
  • Cross-file registration (design amendment, recorded in design.md). connexion registers VAmPI's handlers by operationId in openapi_specs/openapi3.yml; Overleaf registers exportProject in router.mjs. No single-file excerpt can contain that, so a recipe may name context = [...], the harvester writes the stanza per side under <name>/context/<side>/<relpath>, the catalog carries [[pair.context]], and _materialize_side lays the documents out beside the excerpt at their repository-relative paths. mapping._route_manifest_entry_points reads operationId plus the operation's security block; _js_statements joins a registration that spans lines, because reading webRouter.post( alone finds neither the handler nor the middleware that guards it.
  • Anchors. Real-Vuln-Benchmark trap rows take their fixed side from a different, correctly guarded handler of the same snapshot, so handler_context honours line/fix_line and a new fix_function gives the fixed side its registration instead of the vulnerable side's.
  • innermost_function_bounds for brace languages, used only by this mode: function_bounds_at returns the outermost top-level group, which for a handler declared inside a React component is the whole 6000-line component. hunk and enclosing rows keep the bounds they were harvested with.
  • Fetch-then-write. harvest_recipe fetches every blob before the first write, and the driver takes --report so a failed fetch leaves the reviewed excerpt exactly as it was and says why. A half-written re-harvest is indistinguishable from a real upstream change.
  • Bugs the re-harvest exposed and fixed on the way: .mjs and .cjs were not in LANGUAGE_BY_EXTENSION, so the one pair vendored as an ES module had no file target at all — no entry point, no finding, a permanent miss; and _python_route_access matched auth words anywhere in a decorator, so @ns.doc(description='... without proper authorization checks') read the unguarded ThreatByte endpoint as authenticated, silencing every obligation on it. The guard is now the decorator's name, never its prose, and @token_required — the commonest Flask guard — is recognised at all.
  • Corpus corrections: lensops-kubernetesmanagement-fe4910 was labeled setErrorMessage, a React state setter, because the hunk landed in the wrong bounds; it is now loadNamespaces and the pair diff is exactly the fix. Its bug is a browser-side call omitting the header its siblings carry, not a missing guard on an inbound handler, so it carries known_limit = "client_side_caller" and leaves the denominators with a stated reason.
  • Three existing tests encoded the old corpus and now encode the new one: the TypeScript parse pin (the lensops excerpt parses now), the classifier report (labeled 0 -> 4, the four seeded rows; the six title-tier agent rows carry a family but are not labels a maintainer stands behind), and the hygiene report ordering.
  • New honesty finding, recorded in the hygiene artifact as cross_split_duplicates: nine of the ten duplicate groups sit on both sides of the split. Two forks of one project (openniw-jobs-0fc947 train, openniw-jobs-b74de3 holdout) and eight CVEs harvested twice under different names are byte-identical, so the train row teaches the holdout row exactly. split_by_repository cannot see it — the duplicates are different repositories. duplicate_groups is now ordered by name so the committed report does not reshuffle when one excerpt is re-harvested.
  • Pre-existing drift the regeneration corrected, unrelated to this task: catalog.toml in git disagreed with a fresh catalog_gen run at the group-3 tip on two derived labels (origin -> createApplication, normalizeDomain -> getGoogleProvider). Verified in a detached worktree at 7a33756 before touching anything.
• Group 3 (2026-09-06). Tasks 3.1-3.9 implemented in 86ed54c, fcda1ee, e8e6baf, 127dbef, ad7c98e, 346cc70, 16d9bc2, b955251, b8ad42c, 7a33756. RED counts, derived by the reviewer against each task's parent commit: 3.1 4/6, 3.2 6/6, 3.3 6/6, 3.4 3/5, 3.5 10/10, 3.6 12/12, 3.7 9/9, 3.8 7/7, 3.9 7/7. Deviations from the design, recorded here because they were not written down at the time: learning.split duck-types PairCase through getattr rather than importing it; pairs.py reaches learning through function-local imports rather than the design's injected callable; run_baseline takes a scan factory where the design named a client. One flaky test found and fixed in 346cc70 (_is_vulnerable_side keyed on "-v" in str(root), which random temporary directory names matched about once in eight; harness noise recorded as detector noise would inflate every family's regression budget).
• Review round 3 (2026-09-06) APPROVED 3.4; REJECTED 3.1, 3.2, 3.3, 3.5, 3.6, 3.7, 3.8, 3.9. The reviewer ran every check in detached worktrees at the group tip against 71efc9c, derived the RED phase per task, mutation-tested decide, the teacher rule, the snapshot restore and the round's revert paths (7 mutants on the acceptance rule, 6 killed and 1 equivalent), ran the three flakiness-prone files 61 times with varied PYTHONHASHSEED without a failure, and confirmed the three gate outputs byte-identical and the extra-free matrix unchanged. Every rejection was remediated RED-first; the numbers each defect was hiding are recorded below because they are the point of the spec.
  • 3.2, the fixed side was judged against the vulnerable side's line ranges. A fix that adds a guard above the handler moves its first line, and the reviewer measured that 191 of 227 pairs with a resolvable labeled span (84%) have different spans on the two sides. A finding inside the fixed handler was therefore read as outside it: a leak recorded as silence, always in the direction that flatters Youden. score_pair_family now takes fixed_ranges; the fixed side's parse was already being computed and thrown away.
  • 3.2 and 3.9, K runs was declared and never threaded. _evaluate_hunter_pair(..., runs=1) had one call site that passed nothing, evaluate_catalog had no parameter at all, and pairs --k-runs was parsed and discarded; the only test grepped --help. K now flows pairs --k-runs -> evaluate_catalog -> evaluate_pair -> _evaluate_hunter_pair, defaults to three and refuses less, and the test counts scans per side.
  • 3.2, the generalist could never be scored. pairs --hunter fed the class-aware scorer findings with no family: tag, so it reported recall 0 by construction. run_tool_hunter now reads an optional family field into the tag (the design said it would) and make_hunter_scan(taxonomy=...) asks for one from the closed list. A generalist that names no family is unanswerable; one that names a wrong family is counted as noise, which is the honest treatment.
  • 3.3 and 3.9, round zero measured every family with the generalist. run_baseline took one scan, and the CLI handed it the unknown configuration; since related(X, "unknown") is lateral by the taxonomy's own decision that abstention earns no credit, every family except unknown baselined at a structural zero and every noise floor with it. run_baseline now takes a scan_factory and runs each family's own configuration, reports per slice as well as per family, and refuses fewer than three runs. This is the decision the group-2 review asked for on unknown serving as both abstention and generalist: it is the generalist for scanning, and it is never credited as a family for scoring.
  • 3.5, the meta-agent proposer was a stub that returned None after confirming the extra was present, so ousast learning round could never propose anything at any time. It now composes MetaAgent.evolve behind harness_ext against a scratch copy of the family directory — the live tree is never touched, because the round snapshots only after a proposal exists — and builds the proposal from the diff of that copy, so the one-file rule and the lever whitelist are enforced on what the agent did rather than on what it claimed. learning round --proposals FILE runs a recorded sequence offline, which is what makes the command exercisable without the extra at all.
  • 3.6 and 3.1, the holdout refusal existed only as a vocabulary word. journal.py documented refused_holdout and no code path could produce it; the covering test asserted outcome == "refused_holdout" or outcome == "rejected", which passes whichever way the code behaves. The round now runs refuse_if_holdout over every corpus pair the proposal names — in its predictions and in the file it writes — and journals refused_holdout with the pair names. A memory entry that quotes a held-out pair looks exactly like a legitimate hard negative until the holdout number is read as if it meant something.
  • 3.7, the sweep counted absolute failures, not negative flips. Req 9.5 compares against the pre-change state; the code counted every pair of another family that was not pair_correct and compared that to a budget derived from the run-to-run flip rate. On the real corpus, where most pairs fail at round zero, that rejects every round for damage it did not do — a harness that then reports it cannot improve. Unscorable rows voted too, against Req 4.1. The sweep is now scored before and after the change, _negative_flips counts only pairs that were right and are not, unscorable rows leave the count, and attribution is derived from the same comparison instead of from absolute failures.
  • 3.7, cost was metered once, after the minibatch, the holdout and the whole cross-family sweep had all been paid for; Req 9.8 says stop, revert and record. The cap is now read after every stage. A stage that raises is journalled as reverted/stage_failed with its round directory before the exception leaves, because an exception that loses the record also loses the money the round already spent. A rejected round's version is archived: Req 9.7 exists so a configuration that lost overall but is the only one that ever won some pair is not forgotten.
  • 3.8, the corrected lever number was published beside the earlier one, not in place of it (Req 5.3), and the test named for that property never checked it. publish now rewrites the superseded claim to point at the regenerated section, idempotently, and the test asserts the old sentence is gone. Run against the real roadmap: the 2026-09-05 claim "human holdout profile pair_correct 0 -> 6 of 17, Youden 0 -> +0.353" now says it was measured with holdout pairs teaching and names the artifact that corrects it.
  • 3.1, refusals were continue statements. Refusal.degradation() existed and nothing called it. The mechanism lever now records every candidate it declined with the holdout pairs that taught it, leave-one-out records the pairs it kept out of the teaching set, and improve carries both into the round outcome. semantic.seed reported split:train for a train-split identical twin, which is simply untrue; an unscorable row now reports its own reason. evaluate_profiles and evaluate_mechanism_profiles are deliberately split-blind and now say so in their docstrings: they measure a ruleset and never learn from a pair, so a filter there would only shrink the evidence.
  • Test-quality repairs the review asked for, each of which changed a number: two split tests were measuring an x = 1 fixture that teaches nothing whatever the rule says, so both arms now use a pair that really carries a shape and the lever test asserts the same shape is admitted when a train pair taught it; the two CLI scoreboard tests asserted against output that was all zeros, and now use a side-aware detector stub (the shipped scripted stub answers twice, then goes quiet, and puts its finding outside the labeled function — three separate ways to make a family number zero); ScriptedChatClient gained an opt-in cycle so a stub cannot silently run out of answers mid-run; the sandbox-candidate assertion in tests/test_learning_scan.py compared verdict paths against a prefix no implementation would produce and now compares finding ids; the manifest's classifier block reported measure_classifier([]), all zeros, and now reports what this scan actually routed.
  • Carry-forward, unchanged and still capping every holdout number: the nineteen straddling repositories, and — found during task 4.1 — nine of the ten duplicate groups sit on both sides of the split, which the repository-level split cannot see because the duplicates are different repositories.
• Task 4.2 (2026-09-06, RED first: 2 failing tests in tests/test_fixture_difficulty.py, and the check's own arm — test_the_check_catches_a_fixture_that_is_easier_on_one_axis — fails if the fixture hardening is reverted). Two parts:
  • The five twins declare their limit. A content hash is evidence; the catalog row is the maintainer standing behind it, and without one a re-harvest that changes a byte turns a known-unscorable row back into a scored one with no review. A declared limit that names one of the vocabulary's own reasons keeps that spelling, so identical_twin declared and identical_twin computed stay one fact instead of splitting across two buckets in every aggregation. semantic.seed._skip_reason was reordered for the same reason: an unscorable row reports unscorable:<reason> before either the known-limit or the split branch can claim it. Consequence, recorded because it moves a published number: leave-one-out over vibe-py now runs on 30 rows, not 35 — a pair whose two sides are the same code cannot be held out against itself.
  • The fixture-difficulty check (learning/difficulty.py), on three axes the corpus really varies on: binding hops from the identity source to the constrained use, registration distance (0 decorator on the handler, 1 statement elsewhere in the file, 2 another document), and guard distance (0 decorator, 1 call in the body, 2 nothing visible). The comparison is per axis over the whole set, because a suite covers a bug shape with several handlers between them. Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2). Every shared handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no chain was longer than one binding — the checker was passing on a simpler world than the one it meets. from_spec was added to the fixture: the connexion shape those rows really have, a token validated by a call rather than a decorator and five bindings between the token and the constraint. The verdict is committed as benchmarks/measurements/2026-09-06-fixture-difficulty.json with both reaches and the history of the first measurement. One existing expectation moved with it (from_spec reads the user row and then the book: two real operations).
• Task 4.3 (2026-09-06, RED first: 10 failing tests in tests/test_learning_cutoff.py; then green, 11). The catalog generator fills fix_date from the recipe at the precision the recipe has: a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing -01-01 would turn a year into a claim about a month. A row therefore counts as post-cutoff only when its earliest possible date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it — and over-claiming is the direction that would turn contamination into a result. PairFamilyScore carries the date, aggregate(cutoff=...) reports the same numbers over the post-cutoff rows beside the whole number and never instead of it, undated rows are counted on their own, and publish prints the slice with the cutoff it used. Without a configured cutoff every scoreboard says so rather than inventing one; [learning].cutoff_date has no default because guessing a model's training cutoff and publishing a contamination number from the guess would be worse than reporting none. Corpus dates as harvested: vibe-py all 2026, agent-vfc 67 of 69 in 2026, vfc spread 2014-2026, vfc-js 2014-2021.