Open_Ultra_SAST/.kiro/specs/learning-harness/design.md

Design Document: learning-harness

Overview

Purpose: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.

Users: the maintainer runs round zero, evolve rounds and the publish command; operators read scan findings that now carry a family, a detector version and a verifier outcome; reviewers work the classifier disagreement queue.

Impact: the tool hunter becomes the primary detector and the static machinery becomes tools and verifiers. The pair scorer's hunter path, the exporter, leave-one-out, the mechanism lever and the improve journal change behavior; findings, catalogs and the manifest gain additive fields. Gates stay byte-identical.

Goals

• One trustworthy number per family per slice, with its denominator, taxonomy version, noise floor and cost.
• A round can only be accepted when it helps its target family on held-out pairs and hurts no other family beyond its measured budget.
• Nothing above suspicion without a verifier; verifier code and gold labels outside every editable surface.
• Round zero on the web families over vibe-py and agent-vfc before any learning.

Non-Goals

• Browser-engine, interpreter or renderer bugs; crash oracles for C and C++ (the memory-safety family is measured, never gating).
• New static shape families, new hand-written facts, evidence-ladder or disposition changes, merge gates on non-local slices, model fine-tuning.
• A general router framework or a generic agent platform; the classifier is a function and a detector configuration is a directory.

Boundary Commitments

This Spec Owns

• The family taxonomy file, its loader and its version (1.x).
• Classification of pairs, findings and regions, and the classifier's measurement report and review queue (2.x).
• Class-aware pair scoring: outcomes, directional bias, fixed-FPR view, K-run reliable-change test, negative flips (3.x).
• Unscorable accounting and its reasons; the additive unscorable and family fields on catalog rows (4.x).
• The single teacher-selection rule every learning path calls, and holdout refusal (5.x).
• Per-family detector configurations, their loader, the family detector runner and finding tags (6.1–6.5); the chat-endpoint resolution independent of embeddings (6.6).
• The verifier registry keyed by family, canary oracles, the second-judge step (7.x).
• Round zero, the noise floor, evolve rounds, acceptance, attribution, the archive, the round journal, cost caps (8.x, 9.x).
• The corpus repairs listed in 10.x and the publish command with committed artifacts (11.x).

Out of Boundary

• Pair labels, tiers, splits and known_limit semantics beyond the two additive fields (pair-corpus-honesty).
• Facts, walker, dominance, obligations checker internals; the mechanism store schema; the rule and policy levers' validators.
• The sandbox runner, snippet safety rules and Docker images; this spec adds an oracle to the verdict, not a runner.
• HarnessX internals; this spec composes MetaAgent.evolve and the journal behind harness_ext.
• Harvesting new pairs; only re-harvesting existing absence recipes.

Allowed Dependencies

• pairs, benchmark, semantic.loo, semantic.seed, semantic.mechanisms, improve.evolve, improve.validator, tool_hunter, hunter_tools, provider.openrouter, regress.*, sandbox.*, redaction, harness_ext, config.
• Optional extras only through harness_ext (HarnessX) and semantic.extra (tree-sitter, now including TypeScript); every step degrades to a recorded reason without them.
• Direction: learning.families → learning.classify → learning.scoring → learning.split → learning.detectors and learning.verifiers → learning.rounds → learning.publish and cli. learning.* imports the packages above; nothing outside learning imports learning.rounds or learning.publish. pairs and benchmark never import learning.

Revalidation Triggers

• A change to the family list, a family's CWE set or its verifier (taxonomy version bump).
• A change to the pair outcome vocabulary or the reliable-change test.
• A change to the acceptance rule, the regression-budget definition or the journal schema.
• A change to the chat-endpoint resolution order or the tool-hunter finding shape.
• A change to the sandbox verdict contract (oracle callback) or snippet safety rules.

Architecture

Existing Architecture Analysis

• tool_hunter.run_tool_hunter is a 4-step loop over three curated tools with findings parsed into StaticFinding at suspicion (finding_id = tool-hunter:<path>:<line>, tag tool-hunter). It refuses to emit findings when no tool was called. It has no temperature, budget or context knobs beyond max_steps.
• pairs._hunter_matches_expected keys detection on text tokens and _evaluate_hunter_pair counts every fixed-side finding as a leak under fix_policy = silent; both are replaced on the hunter path only. The overlay and inventory paths and pair_gate are untouched.
• evaluate_profiles, evaluate_mechanism_profiles, propose_mechanism_edits, evaluate_loo and build_pair_signals filter by tier and vendored status but not by split; export_mechanisms seeds from every case it is given.
• OpenRouterChatClient and OpenRouterEmbeddingClient read the same OPENROUTER_BASE_URL; complete_chat always sends temperature: 0 and drops reasoning_content from replies. DeepSeek requires thinking disabled for temperature to apply, reasoning_content replayed on tool-call turns, no /v1 suffix, json_object only, no seed.
• regress.candidate derives a verdict from the exit code; snippets are compile-only templates or the model's proposed_snippet; check_snippet_safety bans sockets, curl and workspace writes, which a canary file in /scratch respects.
• MechanismStore is an append-only JSONL with tombstones and deterministic ids, the pattern reused for the archive and the journal.
• HarnessX 0.1.0 provides MetaAgent.evolve(current_config, trajectories_dir, output_dir) with post-flight checks, JournalEntrySpec(round, label, hypothesis_id, levers, predicted_affected, prose), build_context with precision discounted by unpredicted regressions, and RLTask.task_type routing.

Architecture Pattern & Boundary Map

graph TB
    subgraph Corpus
        Catalog[pair catalogs]
        Loader[pairs loader with unscorable and family fields]
    end
    subgraph Learning
        Families[families taxonomy]
        Classify[classifier three tiers]
        Scoring[class aware pair scorer]
        Split[teachers and holdout refusal]
        Detectors[family detector configs and runner]
        Verifiers[verifier registry canary static none]
        Rounds[round zero and evolve rounds]
        Journal[round journal and archive]
        Publish[publish command]
    end
    subgraph Existing
        Hunter[tool hunter loop]
        Chat[chat endpoint client]
        Regress[regress verdicts and sandbox]
        Levers[exporter loo mechanism lever]
        Meta[HarnessX MetaAgent optional]
    end
    Catalog --> Loader --> Scoring
    Families --> Classify --> Scoring
    Families --> Detectors
    Families --> Verifiers
    Classify --> Detectors
    Split --> Levers
    Split --> Rounds
    Detectors --> Hunter --> Chat
    Verifiers --> Regress
    Scoring --> Rounds
    Detectors --> Rounds
    Verifiers --> Rounds
    Rounds --> Journal
    Rounds --> Meta
    Journal --> Publish
    Scoring --> Publish

Architecture Integration:

• Selected pattern: a learning package layered left to right (families → classify → scoring → split → detectors/verifiers → rounds → publish), composing existing seams rather than replacing them.
• Domain boundaries: the corpus owns labels and rows; learning owns class assignment, scoring, teaching rules, detector configurations, verifier keys and rounds; regress owns execution; harness_ext owns the HarnessX seam.
• Existing patterns preserved: append-only JSONL with tombstones for the archive and journal; HarnessRuntime.run_stage and degradations for every step; ScriptedChatClient for offline tests; FakeSandboxRunner for verifier tests.
• New components rationale: each is one requirement area with one owner; the classifier and scorer are functions, the detector configuration is a directory, the journal is a file.
• Steering compliance: core dependencies = []; recall first under a false-positive ceiling, now per family; gates byte-identical; no model-authored artifact retained without held-out validation.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
Chat endpoint	DeepSeek platform API, OpenAI-compatible, https://api.deepseek.com (no /v1); models deepseek-v4-flash (default), deepseek-v4-pro (second judge, review queue)	Detectors, classifier model tier, proposer, judge	Thinking on by default: the client sends thinking: {"type": "disabled"} for detectors and the classifier so temperature applies; reasoning_content replayed on tool-call turns; response_format json_object with empty-content retry; logprobs requested by the classifier tier only. No seed: determinism is best-effort and absorbed by the noise floor. Pricing (peak, USD/1M): flash 0.44 in / 1.32 out, cache hit 0.014; pro 1.32 / 3.96. Concurrency 2500 (flash).
Embedding endpoint	OpenRouter, openai/text-embedding-3-small, unchanged	Prove-order retrieval	Reads OPENROUTER_* only; DeepSeek has no embeddings endpoint.
Evolve loop	HarnessX 0.1.0 (meta_harness.agent.MetaAgent, meta_harness.journal), optional extra	Proposal generation and attribution context	Behind harness_ext.require_harnessx; a scripted proposer stands in without it.
Parsing	tree-sitter extra + tree-sitter-typescript>=0.23	TypeScript rows become scorable	Optional extra only.
Sandbox	existing Docker CLI runner, images python:3.12-alpine, node:22-alpine	Canary verifiers	Oracle callback added to the verdict; safety rules unchanged.
Storage	JSONL under .openultrasast/learning/ and benchmarks/measurements/	Journal, archive, noise floors, published artifacts	Append-only, committed when published.

File Structure Plan

Directory Structure

src/openultrasast/learning/
├── __init__.py            # public surface: load_families, classify_*, score_pair_family, teachers, run_baseline, run_learning_round, publish
├── families.py            # Family, FamilyTaxonomy, load_families(ruleset/families.toml), version, verifier kind per family
├── classify.py            # Classification, classify_pair/finding/region (tiers), measure_classifier, review queue entries
├── scoring.py             # FamilyOutcome, score_pair_family, FamilyMetrics, directional_bias, negative_flips, reliable_change, fixed_fpr_view, unscorable_reason
├── split.py               # teachers(cases), refuse_if_holdout(candidate_pairs, cases) -> Refusal | None
├── detectors.py           # FamilyConfig, load_family_configs, run_family_detector (whole-file context, curated tools), finding tags
├── endpoint.py            # ChatEndpoint, resolve_chat_endpoint (DeepSeek first, OpenRouter fallback, scripted), DeepSeekChatClient adapter
├── verifiers.py           # Verifier protocol, CanaryVerifier, StaticVerifier, NoVerifier, verifier_for(family), second_judge
├── rounds.py              # run_baseline (round zero, noise floor), run_learning_round (propose, minibatch, holdout, sweep, accept/revert), Attribution
├── proposer.py            # Proposer protocol, ScriptedProposer, HarnessXProposer (MetaAgent.evolve behind harness_ext), failure facts, rejected buffer
├── journal.py             # RoundRecord, LearningJournal (append-only JSONL), Archive (family -> winning config per pair)
└── publish.py             # publish(): regenerate numbers from one command, write roadmap table and measurement artifacts
src/openultrasast/ruleset/families.toml          # closed, versioned taxonomy: id, cwes, mechanisms, verifier, description
src/openultrasast/learning/configs/<family>/     # round-zero detector configuration per family (prompt.md, checklist.md, tools.toml, budget.toml, hard_negatives.jsonl, counterexamples.jsonl)
tests/test_learning_families.py, test_learning_classify.py, test_learning_scoring.py, test_learning_split.py, test_learning_detectors.py, test_learning_endpoint.py, test_learning_verifiers.py, test_learning_rounds.py, test_learning_journal.py, test_learning_publish.py, test_fixture_difficulty.py

Modified Files

• src/openultrasast/pairs.py — PairCase.unscorable: str | None computed at load (identical twin via normalized body hash, known_limit passthrough); duplicate report across pairs before split; split_by_repository helper honoring existing split fields and reporting straddles; _evaluate_hunter_pair delegates matching and leaks to learning.scoring through an injected callable so pairs does not import learning; build_pair_signals gains a split filter.
• src/openultrasast/benchmark.py — ExpectedFinding.family: str | None = None (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so pairs and benchmark never import learning).
• src/openultrasast/tool_hunter.py — run_tool_hunter gains system_prompt, context_files, tools, max_chars and tags parameters with today's values as defaults; finding parsing accepts an optional family field; the reply loop replays reasoning_content.
• src/openultrasast/hunter_tools.py — four new tools beside read_file, grep_repo, find_refs: read_definition(symbol), entry_points(path), flows(path, function), obligations(path, function), each clamped to the repository and returning identifiers and line spans only; HUNTER_TOOLS schemas extended.
• src/openultrasast/provider/openrouter.py — OpenRouterChatClient gains extra_body (thinking, response_format, logprobs) and retains the reasoning_content field on assistant messages; replaying it across tool turns is the hunter loop's job (tool_hunter), not the client's. No behavior change for existing callers.
• src/openultrasast/config.py — [models] gains chat_base_url, chat_api_key_env, judge; [learning] section (families_path, configs_dir, k_runs = 5, round_cost_cap_usd, minibatch, cutoff_date); load_dotenv unchanged, tests inject endpoints explicitly.
• src/openultrasast/semantic/seed.py, semantic/loo.py, improve/evolve.py — teacher selection through learning.split.teachers; refusals recorded; evaluate_loo teaches from train only.
• src/openultrasast/regress/candidate.py, regress/verdict.py — run_regression accepts an optional oracle: Callable[[SandboxResult], bool]; verdict_from_result uses the oracle when given, exit code otherwise.
• src/openultrasast/cli.py — ousast learning classify|score|baseline|round|publish; pairs --hunter uses the family scorer and --k-runs; [models].hunter resolution through learning.endpoint.
• pyproject.toml — tree-sitter-typescript>=0.23 in the semantic extra; src/openultrasast/semantic/extra.py maps typescript/tsx.
• benchmarks/pairs/vibe-py/catalog.toml and recipes.toml — the five identical twins gain known_limit = "identical_twin"; absence rows gain family and mode = "handler_context" (re-harvest is a maintainer step).
• Cross-file registration context (amended during task 4.1). Some frameworks keep the route table out of the handler module: connexion registers by operationId in an OpenAPI document, an Express application registers in a router file. A single-file excerpt of such a handler can never contain its registration, so Req 10.2 is unreachable for those rows without a second file. A recipe may therefore name context = ["<repo-relative path>"]; benchmarks/pairs/harvest.py fetches each document at the same two commits, extracts only the stanza that names the handler (extract_registration_context), and writes it under <name>/context/<side>/<relpath> with its own provenance header and the same redaction. The catalog carries [[pair.context]] rows, PairCase.context_files holds (relpath, vuln, fixed), and _materialize_side copies them beside the excerpt at their repository-relative paths so the entry-point mapper sees the same layout the project has. src/openultrasast/mapping.py gains _route_manifest_entry_points, which reads operationId and the operation's security block out of a YAML or JSON document and attributes the record to the module the operation id points at. Trap pairs whose fixed side is a different, correctly guarded handler carry fix_function so the fixed side gets its registration rather than the vulnerable side's.
• benchmarks/pairs/mechanisms.toml — unchanged; families reference mechanism ids.
• .kiro/steering/roadmap.md — rewritten by publish from artifacts.

System Flows

Class-aware scoring of one pair (hunter path)

sequenceDiagram
    participant S as Scorer
    participant C as Classifier
    participant D as FamilyDetectors
    participant V as Verifier
    S->>C: classify_pair(case)
    C-->>S: families or unknown, tier
    loop K runs
        S->>D: run on vuln side and fixed side
        D-->>S: findings with family tags
    end
    S->>S: detection = family match inside labeled function
    S->>S: silence = no labeled family inside function on fixed side
    S->>V: verify claims that a verifier exists for
    V-->>S: suspicion, static corroboration, or proven
    S-->>S: FamilyOutcome, negative flips, reliable change

Decisions: unscorable pairs short-circuit before any run; K runs share one classification; verifier outcomes never change detection or silence, they change the reported rung.

One evolve round

flowchart TB
    Start[failure facts for target family] --> Propose[proposer drafts one structural change]
    Propose --> Refuse{touches holdout or exceeds length cap}
    Refuse -->|yes| Reject[reject and record]
    Refuse -->|no| Mini[train minibatch of target family]
    Mini --> MiniOk{no decrease}
    MiniOk -->|no| Reject
    MiniOk -->|yes| Hold[target family full holdout]
    Hold --> HoldOk{train and holdout non negative and one strictly positive}
    HoldOk -->|no| Reject
    HoldOk -->|yes| Sweep[every other family holdout]
    Sweep --> SweepOk{negative flips within each budget}
    SweepOk -->|no| Reject
    SweepOk -->|yes| Accept[accept freeze config bump version]
    Accept --> Journal[journal hypothesis predicted attribution cost]
    Reject --> Journal
    Journal --> Buffer[rejected edit buffer updated]

Decisions: the proposer never sees holdout inputs, outputs or scores; cost is metered per stage and the round is reverted when the cap is crossed at any stage; ties reject.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1–1.4	closed taxonomy, verifier per family, version, attributes not keys	Families	load_families, FamilyTaxonomy	—
2.1–2.5	three-tier multi-label classifier, measured, review queue, whole corpus	Classifier	classify_*, measure_classifier	scoring
3.1–3.6	family-in-function detection and silence, outcomes, bias, FPR, K runs, no text tokens	Scorer	score_pair_family, FamilyMetrics	scoring
4.1–4.5	unscorable reasons, never delete, dedup, repository split, denominators shown	Loader (pairs), Scorer, Publish	PairCase.unscorable, unscorable_reason, split_by_repository	—
5.1–5.4	train-only teachers, refusal, re-measure lever, proposer blind	Split, Levers, Proposer	teachers, refuse_if_holdout	round
6.1–6.6	per-family config, frozen, routing plus generalist, whole file, curated tools, independent chat endpoint	Detectors, Endpoint	FamilyConfig, run_family_detector, resolve_chat_endpoint	scoring
7.1–7.6	suspicion unless verified, canary oracles, static corroboration, none, second judge, sealed surface	Verifiers, Regress	Verifier, verifier_for, second_judge, run_regression(oracle=)	scoring
8.1–8.4	clone, K = 5 noise floor, publish round zero, web families first	Rounds	run_baseline, NoiseFloor	—
9.1–9.9	failure facts, one change, journal, staged evaluation, acceptance, attribution, archive, cost cap, retention	Rounds, Proposer, Journal	run_learning_round, Proposer, RoundRecord, Archive	round
10.1–10.5	TypeScript, re-harvest, identical twins, fixture difficulty, post-cutoff slice	Extra, Corpus, Tests	semantic.extra, catalog fields, test_fixture_difficulty	—
11.1–11.5	one command, artifacts committed, trajectories persisted, degrade with reason, gates identical	Publish, Journal, Runtime	publish, LearningJournal	—

Components and Interfaces

Component	Layer	Intent	Req	Key Dependencies	Contracts
Families	data	closed taxonomy with verifier per family	1.x	ruleset file (P0)	Service
Classifier	logic	assign families with tier, measure itself	2.x	Families (P0), Endpoint (P1)	Service, State
Scorer	logic	class-aware pair outcomes and statistics	3.x, 4.x	Families (P0); findings from Detectors and rungs from Verifiers are inputs, not imports (P1)	Service
Split	logic	the one teacher rule and holdout refusal	5.x	pairs (P0)	Service
Detectors	logic	per-family configuration and runner	6.1–6.5	tool_hunter (P0), Endpoint (P0), Classifier (P0)	Service, State
Endpoint	integration	independent chat endpoint resolution and DeepSeek adapter	6.6	provider.openrouter (P0), config (P0)	Service
Verifiers	integration	family-keyed oracles and the second judge	7.x	regress, sandbox (P0), obligations (P1)	Service
Rounds	orchestration	round zero, noise floor, evolve rounds, acceptance	8.x, 9.x	Scorer, Detectors, Proposer, Journal (P0)	Batch, State
Proposer	integration	structural change proposals from failure facts	9.1–9.2	harness_ext (P1), Endpoint (P1)	Service
Journal	data	round records, attribution, archive	9.3, 9.6, 9.7, 11.3	JSONL (P0)	State
Publish	orchestration	numbers from one command, artifacts, roadmap	11.x	Scorer, Journal (P0)	Batch

Data layer

Families

Field	Detail
Intent	Load and version the closed family taxonomy with its CWE set, mechanism set and verifier kind
Requirements	1.1, 1.2, 1.3, 1.4

Responsibilities & Constraints

• Exactly ten families as decided; ids are stable identifiers; the file carries version.
• Verifier kind per family is one of canary, static, none; memory safety is none.
• No component may route or score on a CWE or mechanism id; they are looked up only through a family.

Dependencies: Inbound: Classifier, Scorer, Detectors, Verifiers (P0). External: ruleset/families.toml (P0).

Contracts: Service [x]

@dataclass(frozen=True)
class Family:
    id: str                      # e.g. "injection", "path", "deserialization", "access_control", "output_encoding", "untrusted_destination", "prototype", "config_secrets", "memory", "unknown"
    cwes: frozenset[str]
    mechanisms: frozenset[str]
    verifier: Literal["canary", "static", "none"]
    description: str

@dataclass(frozen=True)
class FamilyTaxonomy:
    version: str
    families: tuple[Family, ...]
    def by_id(self, family_id: str) -> Family: ...
    def family_of_cwe(self, cwe: str) -> Family | None: ...
    def family_of_mechanism(self, mechanism: str) -> Family | None: ...
    def related(self, a: str, b: str) -> Literal["same", "parent_child", "lateral", "fabricated"]: ...

def load_families(path: Path | None = None) -> FamilyTaxonomy   # raises FamiliesError naming the field on any deviation

• Invariant: unknown is always present and never has a verifier.

Journal

Field	Detail
Intent	Append-only record of rounds with attribution, and the archive of which configuration wins which pair
Requirements	9.3, 9.6, 9.7, 11.3

Contracts: State [x]

@dataclass(frozen=True)
class RoundRecord:
    round: int
    family: str
    hypothesis: str
    levers: tuple[str, ...]           # "tool" | "checklist" | "counterexample" | "memory" | "prompt"
    predicted_affected: tuple[str, ...]
    predicted_at_risk: tuple[str, ...]
    outcome: Literal["accepted", "rejected", "reverted_cost", "refused_holdout"]
    reason: str
    target_train_delta: float
    target_holdout_delta: float
    sweep: dict[str, int]             # family -> negative flips
    attribution: Attribution          # flipped_predicted, flipped_unpredicted, precision
    cost_usd: float
    taxonomy_version: str
    config_version: str

class LearningJournal:                 # .openultrasast/learning/journal.jsonl
    def append(self, record: RoundRecord) -> None: ...
    def rounds(self) -> tuple[RoundRecord, ...]: ...
    def rejected_buffer(self, family: str) -> tuple[str, ...]: ...   # hypotheses rejected for this family

class Archive:                          # .openultrasast/learning/archive.jsonl
    def record(self, family: str, config_version: str, pair: str, outcome: FamilyOutcome) -> None: ...
    def winners(self, family: str) -> dict[str, str]: ...             # pair -> config version that scores it best

• Persistence: JSONL, one record per line, never rewritten; the HarnessX journal is mirrored when the extra is present, never the source of truth.

Logic layer

Classifier

Field	Detail
Intent	Assign zero or more families with the deciding tier; measure itself; queue disagreements
Requirements	2.1, 2.2, 2.3, 2.4, 2.5

Responsibilities & Constraints

• Tier 1 (deterministic): rule id, sink family from facts, obligation kind, mechanism id, labeled CWE; any hit decides without a model.
• Tier 2 (model): json_object question with the taxonomy definitions, thinking disabled, logprobs requested; averaged log-probability under a threshold yields unknown.
• Tier 3: unknown.
• Never writes a label; disagreements go to .openultrasast/learning/review-queue.jsonl.
• Validates every loaded catalog family label against the taxonomy at classification time and rejects a fabricated id by name; the corpus loader stores the label as an opaque string.

Dependencies: Inbound: Scorer, Detectors (P0). Outbound: Families (P0), Endpoint (P1, degrades to tier 1 + unknown without a client).

Contracts: Service [x] State [x]

@dataclass(frozen=True)
class Classification:
    families: tuple[str, ...]          # () means unknown
    tier: Literal["static", "model", "none"]
    confidence: float | None

def classify_pair(case: PairCase, taxonomy: FamilyTaxonomy, *, client: ChatClient | None) -> Classification
def classify_finding(finding: StaticFinding, taxonomy: FamilyTaxonomy, *, client: ChatClient | None) -> Classification
def classify_region(path: str, text: str, function: str | None, taxonomy: FamilyTaxonomy, *, facts, client: ChatClient | None) -> Classification

@dataclass(frozen=True)
class ClassifierReport:
    taxonomy_version: str
    agreement: float                   # hierarchical credit vs maintainer labels on reviewed pairs
    confusion: dict[str, dict[str, int]]
    random_router: float
    oracle_router: float
    per_tier: dict[str, int]
    unknown: int
    queued: int

def measure_classifier(cases: Sequence[PairCase], taxonomy: FamilyTaxonomy, *, client: ChatClient | None) -> ClassifierReport

• Postcondition: every reviewed pair whose label disagrees with the classification appears once in the review queue with both answers and the tier.

Scorer

Field	Detail
Intent	Score one pair per family from detector findings, aggregate per family and slice, decide reliable change
Requirements	3.1–3.6, 4.1, 4.5

Responsibilities & Constraints

• Detection: a finding whose family: tag relates same or parent_child to the labeled family and whose line lies in the labeled function; fabricated scores the maximum penalty; text is never inspected.
• Silence: no same/parent_child finding of the labeled family inside the labeled function on the fixed side; other findings are counted in other_findings, never as leaks.
• K runs per pair at every evaluation (baseline K = 5, rounds K ≥ 3 as 3.5 requires); reliable change against the last blessed round via a paired test on per-pair outcomes; negative flips per family.

Contracts: Service [x]

FamilyOutcome = Literal["pair_correct", "both_flagged", "both_silent", "reversed", "unscorable"]

@dataclass(frozen=True)
class PairFamilyScore:
    pair: str
    family: str
    runs: tuple[FamilyOutcome, ...]    # length K
    outcome: FamilyOutcome             # majority of runs
    other_findings_vuln: int
    other_findings_fixed: int
    rung: Literal["suspicion", "static_corroboration", "proven"]
    unscorable_reason: str | None

@dataclass(frozen=True)
class FamilyMetrics:
    family: str
    scorable: int
    unscorable: dict[str, int]         # reason -> count
    outcomes: dict[FamilyOutcome, int]
    recall: float
    silence: float
    youden: float
    directional_bias: float            # (both_flagged - both_silent) / scorable
    fixed_fpr_recall: float            # recall at the family's FPR ceiling
    negative_flips: int                # vs last blessed round
    reliable_change: bool
    taxonomy_version: str

def score_pair_family(case: PairCase, family: str, vuln_runs: Sequence[Sequence[StaticFinding]], fixed_runs: Sequence[Sequence[StaticFinding]], *, ranges, taxonomy) -> PairFamilyScore
def aggregate(scores: Sequence[PairFamilyScore], *, baseline: Sequence[PairFamilyScore] | None) -> dict[str, FamilyMetrics]
def unscorable_reason(case: PairCase, *, parse_ok: bool, ranges) -> str | None   # identical_twin | unsupported_language | unresolved_label | missing_context | known_limit:<x>

Split

Field	Detail
Intent	The one rule that selects teachers, and the refusal for anything touching holdout
Requirements	5.1, 5.2, 5.3, 5.4

Contracts: Service [x]

def teachers(cases: Sequence[PairCase]) -> tuple[PairCase, ...]                 # vendored, gating tier, split == "train", scorable
def refuse_if_holdout(taught_by: Iterable[str], cases: Sequence[PairCase]) -> Refusal | None   # Refusal(pairs=..., reason="holdout_pair")

• export_mechanisms, evaluate_loo, evaluate_profiles, evaluate_mechanism_profiles, propose_mechanism_edits and build_pair_signals obtain teachers only through teachers; refusals are recorded as degradations {"stage": "learning", "reason": "holdout_pair_refused", "pairs": [...]}.

Detectors

Field	Detail
Intent	Load frozen per-family configurations and run the family detector with whole-file context and curated tools
Requirements	6.1, 6.2, 6.3, 6.4, 6.5

Responsibilities & Constraints

• A configuration is a directory learning/configs/<family>/ with prompt.md (length cap enforced at load), checklist.md, tools.toml (subset of read_file, grep_repo, find_refs, read_definition, entry_points, flows, obligations), budget.toml (max_steps, max_cost_usd, max_chars), hard_negatives.jsonl, counterexamples.jsonl, version.
• Round zero writes every family's directory from the current hunter prompt; a round rewrites only its target family and bumps that version.
• The runner builds the user message from the whole file plus resolved definitions, runs run_tool_hunter with the family's tools and prompt, and tags findings family:<id>, detector:<id>@<version>; the generalist is the unknown family's configuration.

Contracts: Service [x] State [x]

@dataclass(frozen=True)
class FamilyConfig:
    family: str
    version: str
    prompt: str
    checklist: str
    tools: tuple[str, ...]
    max_steps: int
    max_cost_usd: float
    max_chars: int
    hard_negatives: tuple[dict[str, object], ...]
    counterexamples: tuple[dict[str, object], ...]

def load_family_configs(configs_dir: Path, taxonomy: FamilyTaxonomy) -> dict[str, FamilyConfig]   # raises ConfigError on cap or tool violations
def run_family_detector(root: Path, region: Region, config: FamilyConfig, *, client: ChatClient, model: str) -> list[StaticFinding]

• Precondition: region.families from the classifier; postcondition: every finding carries exactly one family: tag and one detector: tag.

Endpoint

Field	Detail
Intent	Resolve the chat endpoint independently of the embedding endpoint and adapt DeepSeek quirks
Requirements	6.6

Responsibilities & Constraints

• Resolution order: explicit injection (tests) → OPENULTRASAST_HUNTER_CLIENT scripted modes → DEEPSEEK_API_KEY with DEEPSEEK_BASE_URL (default https://api.deepseek.com) → OPENROUTER_API_KEY with OPENROUTER_BASE_URL. Embeddings keep reading OPENROUTER_* only.
• [models]: hunter (default deepseek-v4.1-flash), judge (default deepseek-v4.1-flash), chat_base_url, chat_api_key_env. V4 Pro was discontinued on 2026-09-10 and its requests route to V4.1 Flash from 2026-09-14, billed at Flash's price. Detector and judge are now the SAME model, so any claim that the judge is a stronger second opinion is a claim about the prompt and the evidence put to it, not about model class.
• Adapter: thinking disabled and temperature 0 for detectors and classifier; reasoning_content replayed; response_format json_object with one retry on empty content; logprobs on request; usage fields normalized (prompt_cache_hit_tokens) into cost accounting.

Contracts: Service [x]

@dataclass(frozen=True)
class ChatEndpoint:
    base_url: str
    api_key: str
    provider: Literal["deepseek", "openrouter", "scripted"]
    thinking: bool
    price_in_per_m: float
    price_out_per_m: float

def resolve_chat_endpoint(config: ResolvedConfig, *, override: ChatClient | None = None) -> tuple[ChatClient, ChatEndpoint] | None
class DeepSeekChatClient:   # implements ChatClient; wraps OpenRouterChatClient with extra_body and reasoning_content replay
    def complete(self, *, model, messages, tools, timeout_seconds=60, json_object=False, logprobs=False) -> ChatResponse

Verifiers

Field	Detail
Intent	Confirm a claim through the family's oracle, or refuse to raise it
Requirements	7.1–7.6

Responsibilities & Constraints

• CanaryVerifier (injection, path, deserialization, and command execution inside injection): writes a snippet from a family template that imports the target function, plants a canary (row in an in-memory database, file under /scratch, object with a marker method) and prints CANARY:<token> only when the vulnerability fires; the oracle is the token in stdout, never the exit code. The snippet passes check_snippet_safety unchanged.
• StaticVerifier (access control): the obligations checker's finding on the same function raises to static_corroboration.
• NoVerifier: stays at suspicion.
• second_judge: a deepseek-v4.1-flash call with the snippet, the oracle output and the claim, answering json_object {"confirms": bool, "reason": str}; a proven claim is published only when it confirms.
• Templates, oracle code and gold labels live under learning/verifiers/ and benchmarks/, both excluded from every proposer write root.

Contracts: Service [x]

class Verifier(Protocol):
    kind: Literal["canary", "static", "none"]
    def verify(self, claim: StaticFinding, *, root: Path, language: str, runner: SandboxRunner, limits: SandboxConfig) -> Verification

@dataclass(frozen=True)
class Verification:
    rung: Literal["suspicion", "static_corroboration", "proven"]
    oracle_output: str
    reason: str

def verifier_for(family: Family) -> Verifier
def second_judge(verification: Verification, claim: StaticFinding, *, client: ChatClient, model: str) -> bool

• regress.run_regression(..., oracle=Callable[[SandboxResult], bool] | None): when given, TRIGGERABLE iff the oracle returns true; exit-code behavior unchanged otherwise.

Orchestration layer

Rounds

Field	Detail
Intent	Round zero with noise floors; evolve rounds with staged evaluation, acceptance, attribution and revert
Requirements	8.1–8.4, 9.1–9.9

Responsibilities & Constraints

• run_baseline: clone the current hunter prompt into every family directory, run K = 5 over scorable vendored pairs of the selected slices, compute per-family negative-flip rate between runs as NoiseFloor, persist noise-floors.json, publish round zero. Artifacts and floors are keyed by detector model (baseline/<model>/), so a second model is baselined by changing [models].hunter and re-running; publish renders a model comparison table from every baseline present. This is the maintainer's reconsideration path if the default model underperforms; configurations are never touched by a model swap.
• run_learning_round: failure facts for the target family → proposer → refusal checks (holdout, length cap, allowed levers) → minibatch → holdout → sweep → accept (freeze, bump version, archive) or revert (restore the directory snapshot) → journal → rejected buffer.
• Acceptance: train_delta >= 0 and holdout_delta >= 0 and (train_delta > 0 or holdout_delta > 0) and all(sweep[f] <= floor[f]); ties reject.
• Cost metered per stage from usage; crossing round_cost_cap_usd at any stage reverts and records reverted_cost.
• The proposer receives no holdout material; scoring of holdout happens after the proposal is frozen.

Contracts: Batch [x] State [x]

@dataclass(frozen=True)
class NoiseFloor:
    family: str
    k_runs: int
    negative_flip_rate: float
    budget_flips: int                  # ceil(rate * scorable pairs)

def run_baseline(cases, *, slices, taxonomy, configs_dir, client, model, k_runs=5, out_dir) -> BaselineReport
def run_learning_round(cases, *, family, taxonomy, configs_dir, proposer, client, model, journal, archive, floors, cost_cap_usd, minibatch) -> RoundRecord

• Idempotency: a round directory is written under .openultrasast/learning/rounds/<n>/ with the snapshot, proposal, scores and trajectories; rerunning a round number refuses.
• Every stage scores with K runs as the scorer requires; the noise floor is the tolerance for the cross-family sweep, not a substitute for repeated runs.

Proposer

Field	Detail
Intent	Draft one structural change for one family from structured failure facts
Requirements	9.1, 9.2, 5.4

Contracts: Service [x]

@dataclass(frozen=True)
class FailureFacts:
    family: str
    misses: tuple[MissFact, ...]       # pair, function, sink_or_check, line, verifier_outcome, runs
    leaks: tuple[LeakFact, ...]
    rejected: tuple[str, ...]          # hypotheses from the buffer
    config: FamilyConfig

@dataclass(frozen=True)
class Proposal:
    hypothesis: str
    lever: Literal["tool", "checklist", "counterexample", "memory", "prompt"]
    change: dict[str, object]          # file-level edit inside the family directory
    predicted_affected: tuple[str, ...]
    predicted_at_risk: tuple[str, ...]

class Proposer(Protocol):
    def propose(self, facts: FailureFacts) -> Proposal | None

class ScriptedProposer: ...            # tests and offline runs
class HarnessXProposer: ...            # MetaAgent.evolve with allowed_write_roots = the family directory only; degrades to None with a recorded reason without the extra

Publish

Field	Detail
Intent	Regenerate every published number from one command and commit its artifacts
Requirements	11.1, 11.2, 4.5

Contracts: Batch [x]

def publish(cases, *, taxonomy, journal, floors, out_dir=Path("benchmarks/measurements"), roadmap=Path(".kiro/steering/roadmap.md")) -> PublishReport

• Writes <date>-<slice>-<family>.json artifacts, a markdown table with scorable/unscorable counts, taxonomy version, noise floor and cost per point beside every number, and replaces the roadmap's measured section between markers.

Data Models

Domain Model

• Aggregates: FamilyTaxonomy (versioned, immutable per version); FamilyConfig (per family, versioned, frozen between rounds); RoundRecord (append-only); PairCase (owned by the corpus, gains two additive fields).
• Invariants: a finding carries exactly one family: tag; a pair has at most one unscorable reason; a round targets exactly one family; the archive maps a pair to one winning configuration per family.

Data Contracts

• Catalog row additions: [[pair.expected]] family = "<id>" (opaque at load, validated by the Classifier), known_limit = "identical_twin" for the five twins, fix_date = "YYYY-MM-DD" populated by the catalog generator from recipe metadata where available (rows without it are counted as undated in the post-cutoff slice); unscorable is computed, never stored.
• Manifest block learning = {taxonomy_version, families: {id: FamilyMetrics}, classifier: ClassifierReport, round: n | null}; findings carry tags family:, detector:, verifier:<rung>.
• Journal and archive JSONL as above; noise-floors.json; review-queue.jsonl entries {pair, label_family, classified, tier, confidence}.

Error Handling

• Taxonomy or configuration deviations fail fast at load with the field named (FamiliesError, ConfigError).
• Missing extras degrade with a recorded reason: no HarnessX → ScriptedProposer or proposer_unavailable; no TypeScript grammar → rows unsupported_language; no sandbox → NoVerifier for canary families with sandbox_unavailable.
• Endpoint errors: transient retries as today; empty json_object content retried once then recorded; a round whose endpoint fails mid-stage is reverted with endpoint_failed.
• Holdout refusal, cost overrun and safety rejection are outcomes, not exceptions, and always land in the journal.
• Monitoring: every stage runs under HarnessRuntime.run_stage; cost per stage and per round is written to the round directory.

Testing Strategy

• Unit: taxonomy loader rejects an eleventh family and a missing verifier kind (1.1, 1.2); related returns parent_child for a CWE child and fabricated for an unknown id (3.1); classifier tier 1 decides from a rule id without a client and tier 2 falls to unknown under the threshold with a scripted client (2.1, 2.2); scorer counts detection only inside the labeled function and never from text (3.1, 3.6); silence ignores other-family findings (3.2); directional bias and negative flips on synthetic run sets (3.3, 3.5); unscorable_reason on an identical twin and an unsupported language (4.1); teachers excludes holdout and refuse_if_holdout names the pair (5.1, 5.2); configuration loader rejects a prompt over the cap and a tool outside the allowed set (6.1, 6.5); endpoint resolution prefers DeepSeek over OpenRouter and never touches the embedding variables (6.6); canary snippet passes safety and the oracle keys on the token (7.2); acceptance rule rejects ties and budget breaches (9.5); journal round-trips a record and the rejected buffer (9.3, 9.6).
• Integration: pairs --hunter with a scripted client on the shared handler fixture yields family outcomes and the manifest block (3.x, 6.3); export_mechanisms and evaluate_loo teach from train only and record refusals (5.1, 5.3); run_baseline with a scripted client writes noise floors and round-zero artifacts (8.x); run_learning_round with ScriptedProposer accepts a helpful change, rejects a tie, reverts on cost and journals attribution (9.x); run_regression(oracle=) returns TRIGGERABLE on the token with FakeSandboxRunner (7.2).
• Corpus: TypeScript rows parse under the extra (10.1); the five twins load as known_limit:identical_twin and are excluded from denominators (10.3, 4.1); test_fixture_difficulty fails on a fixture easier than its named rows (10.4); post-cutoff slice reported separately (10.5).
• Gates: gate, map_gate, pair_gate stdout byte-identical to the pre-feature commit; extra-free suite green with semantic and harnessx tests skipping (11.5).
• Live (maintainer, not CI): round zero on vibe-py and agent-vfc web families through DeepSeek, artifacts committed under benchmarks/measurements/.

Security Considerations

• Every prompt and every persisted trajectory passes redact_secrets; verifier templates, oracle code and gold labels are outside the proposer's write roots; snippets keep today's safety bans; the sandbox stays network-off and workspace read-only; the second judge sees the snippet and oracle output, never the labels.

Performance & Cost

• Round zero: K = 5 over ~250 vendored pairs, both sides, whole-file context ≈ 25M input tokens; at DeepSeek flash peak pricing under $15, off-peak half. A round: minibatch + holdout + sweep ≈ one baseline pass; round_cost_cap_usd default 10. Concurrency bounded by the runtime's existing budget controls.

Migration Strategy

• Phase 1 (offline, no model): taxonomy, classifier tier 1, scorer, split, unscorable rows, endpoint resolution, TypeScript grammar, identical twins, fixture-difficulty check; existing numbers re-published with denominators.
• Phase 2 (live): round zero on web families; noise floors; classifier tier 2 measured; review queue worked.
• Phase 3: verifiers for the four canary families and the static verifier; second judge.
• Phase 4: evolve rounds with ScriptedProposer first, then HarnessXProposer; obligations lever as the access-control family's first round.
• Rollback: every round is a directory snapshot; publish regenerates from artifacts, so a bad round is reverted by restoring the family directory and re-publishing.