**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
**Users**: the maintainer runs round zero, evolve rounds and the publish command; operators read scan findings that now carry a family, a detector version and a verifier outcome; reviewers work the classifier disagreement queue.
**Impact**: the tool hunter becomes the primary detector and the static machinery becomes tools and verifiers. The pair scorer's hunter path, the exporter, leave-one-out, the mechanism lever and the improve journal change behavior; findings, catalogs and the manifest gain additive fields. Gates stay byte-identical.
### Goals
- One trustworthy number per family per slice, with its denominator, taxonomy version, noise floor and cost.
- A round can only be accepted when it helps its target family on held-out pairs and hurts no other family beyond its measured budget.
- Nothing above `suspicion` without a verifier; verifier code and gold labels outside every editable surface.
- Round zero on the web families over vibe-py and agent-vfc before any learning.
### Non-Goals
- Browser-engine, interpreter or renderer bugs; crash oracles for C and C++ (the memory-safety family is measured, never gating).
- New static shape families, new hand-written facts, evidence-ladder or disposition changes, merge gates on non-local slices, model fine-tuning.
- A general router framework or a generic agent platform; the classifier is a function and a detector configuration is a directory.
## Boundary Commitments
### This Spec Owns
- The family taxonomy file, its loader and its version (1.x).
- Classification of pairs, findings and regions, and the classifier's measurement report and review queue (2.x).
- Unscorable accounting and its reasons; the additive `unscorable` and `family` fields on catalog rows (4.x).
- The single teacher-selection rule every learning path calls, and holdout refusal (5.x).
- Per-family detector configurations, their loader, the family detector runner and finding tags (6.1–6.5); the chat-endpoint resolution independent of embeddings (6.6).
- The verifier registry keyed by family, canary oracles, the second-judge step (7.x).
- Round zero, the noise floor, evolve rounds, acceptance, attribution, the archive, the round journal, cost caps (8.x, 9.x).
- The corpus repairs listed in 10.x and the publish command with committed artifacts (11.x).
### Out of Boundary
- Pair labels, tiers, splits and `known_limit` semantics beyond the two additive fields (`pair-corpus-honesty`).
- Facts, walker, dominance, obligations checker internals; the mechanism store schema; the rule and policy levers' validators.
- The sandbox runner, snippet safety rules and Docker images; this spec adds an oracle to the verdict, not a runner.
- HarnessX internals; this spec composes `MetaAgent.evolve` and the journal behind `harness_ext`.
- Harvesting new pairs; only re-harvesting existing absence recipes.
- Optional extras only through `harness_ext` (HarnessX) and `semantic.extra` (tree-sitter, now including TypeScript); every step degrades to a recorded reason without them.
- Direction: `learning.families` → `learning.classify` → `learning.scoring` → `learning.split` → `learning.detectors` and `learning.verifiers` → `learning.rounds` → `learning.publish` and `cli`. `learning.*` imports the packages above; nothing outside `learning` imports `learning.rounds` or `learning.publish`. `pairs` and `benchmark` never import `learning`.
### Revalidation Triggers
- A change to the family list, a family's CWE set or its verifier (taxonomy version bump).
- A change to the pair outcome vocabulary or the reliable-change test.
- A change to the acceptance rule, the regression-budget definition or the journal schema.
- A change to the chat-endpoint resolution order or the tool-hunter finding shape.
- A change to the sandbox verdict contract (oracle callback) or snippet safety rules.
## Architecture
### Existing Architecture Analysis
-`tool_hunter.run_tool_hunter` is a 4-step loop over three curated tools with findings parsed into `StaticFinding` at `suspicion` (`finding_id = tool-hunter:<path>:<line>`, tag `tool-hunter`). It refuses to emit findings when no tool was called. It has no temperature, budget or context knobs beyond `max_steps`.
-`pairs._hunter_matches_expected` keys detection on text tokens and `_evaluate_hunter_pair` counts every fixed-side finding as a leak under `fix_policy = silent`; both are replaced on the hunter path only. The overlay and inventory paths and `pair_gate` are untouched.
-`evaluate_profiles`, `evaluate_mechanism_profiles`, `propose_mechanism_edits`, `evaluate_loo` and `build_pair_signals` filter by tier and vendored status but not by split; `export_mechanisms` seeds from every case it is given.
-`OpenRouterChatClient` and `OpenRouterEmbeddingClient` read the same `OPENROUTER_BASE_URL`; `complete_chat` always sends `temperature: 0` and drops `reasoning_content` from replies. DeepSeek requires thinking disabled for temperature to apply, `reasoning_content` replayed on tool-call turns, no `/v1` suffix, `json_object` only, no seed.
-`regress.candidate` derives a verdict from the exit code; snippets are compile-only templates or the model's `proposed_snippet`; `check_snippet_safety` bans sockets, curl and workspace writes, which a canary file in `/scratch` respects.
-`MechanismStore` is an append-only JSONL with tombstones and deterministic ids, the pattern reused for the archive and the journal.
- HarnessX 0.1.0 provides `MetaAgent.evolve(current_config, trajectories_dir, output_dir)` with post-flight checks, `JournalEntrySpec(round, label, hypothesis_id, levers, predicted_affected, prose)`, `build_context` with precision discounted by unpredicted regressions, and `RLTask.task_type` routing.
### Architecture Pattern & Boundary Map
```mermaid
graph TB
subgraph Corpus
Catalog[pair catalogs]
Loader[pairs loader with unscorable and family fields]
end
subgraph Learning
Families[families taxonomy]
Classify[classifier three tiers]
Scoring[class aware pair scorer]
Split[teachers and holdout refusal]
Detectors[family detector configs and runner]
Verifiers[verifier registry canary static none]
Rounds[round zero and evolve rounds]
Journal[round journal and archive]
Publish[publish command]
end
subgraph Existing
Hunter[tool hunter loop]
Chat[chat endpoint client]
Regress[regress verdicts and sandbox]
Levers[exporter loo mechanism lever]
Meta[HarnessX MetaAgent optional]
end
Catalog --> Loader --> Scoring
Families --> Classify --> Scoring
Families --> Detectors
Families --> Verifiers
Classify --> Detectors
Split --> Levers
Split --> Rounds
Detectors --> Hunter --> Chat
Verifiers --> Regress
Scoring --> Rounds
Detectors --> Rounds
Verifiers --> Rounds
Rounds --> Journal
Rounds --> Meta
Journal --> Publish
Scoring --> Publish
```
**Architecture Integration**:
- Selected pattern: a `learning` package layered left to right (families → classify → scoring → split → detectors/verifiers → rounds → publish), composing existing seams rather than replacing them.
- Domain boundaries: the corpus owns labels and rows; `learning` owns class assignment, scoring, teaching rules, detector configurations, verifier keys and rounds; `regress` owns execution; `harness_ext` owns the HarnessX seam.
- Existing patterns preserved: append-only JSONL with tombstones for the archive and journal; `HarnessRuntime.run_stage` and degradations for every step; `ScriptedChatClient` for offline tests; `FakeSandboxRunner` for verifier tests.
- New components rationale: each is one requirement area with one owner; the classifier and scorer are functions, the detector configuration is a directory, the journal is a file.
- Steering compliance: core `dependencies = []`; recall first under a false-positive ceiling, now per family; gates byte-identical; no model-authored artifact retained without held-out validation.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
| Chat endpoint | DeepSeek platform API, OpenAI-compatible, `https://api.deepseek.com` (no `/v1`); models `deepseek-v4-flash` (default), `deepseek-v4-pro` (second judge, review queue) | Detectors, classifier model tier, proposer, judge | Thinking on by default: the client sends `thinking: {"type": "disabled"}` for detectors and the classifier so `temperature` applies; `reasoning_content` replayed on tool-call turns; `response_format json_object` with empty-content retry; `logprobs` requested by the classifier tier only. No seed: determinism is best-effort and absorbed by the noise floor. Pricing (peak, USD/1M): flash 0.44 in / 1.32 out, cache hit 0.014; pro 1.32 / 3.96. Concurrency 2500 (flash). |
| Embedding endpoint | OpenRouter, `openai/text-embedding-3-small`, unchanged | Prove-order retrieval | Reads `OPENROUTER_*` only; DeepSeek has no embeddings endpoint. |
| Evolve loop | HarnessX 0.1.0 (`meta_harness.agent.MetaAgent`, `meta_harness.journal`), optional extra | Proposal generation and attribution context | Behind `harness_ext.require_harnessx`; a scripted proposer stands in without it. |
| Parsing | tree-sitter extra + `tree-sitter-typescript>=0.23` | TypeScript rows become scorable | Optional extra only. |
-`src/openultrasast/pairs.py`: `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
-`src/openultrasast/benchmark.py`: `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
-`src/openultrasast/tool_hunter.py`: `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
-`src/openultrasast/hunter_tools.py`: four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
-`src/openultrasast/provider/openrouter.py`: `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
-`src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py`: teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
-`src/openultrasast/regress/candidate.py`, `regress/verdict.py`: `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
-`src/openultrasast/cli.py`: `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
-`pyproject.toml`: `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
-`benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml`: the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
- **Cross-file registration context (amended during task 4.1).** Some frameworks keep the route table out of the handler module: connexion registers by `operationId` in an OpenAPI document, an Express application registers in a router file. A single-file excerpt of such a handler can never contain its registration, so Req 10.2 is unreachable for those rows without a second file. A recipe may therefore name `context = ["<repo-relative path>"]`; `benchmarks/pairs/harvest.py` fetches each document at the same two commits, extracts only the stanza that names the handler (`extract_registration_context`), and writes it under `<name>/context/<side>/<relpath>` with its own provenance header and the same redaction. The catalog carries `[[pair.context]]` rows, `PairCase.context_files` holds `(relpath, vuln, fixed)`, and `_materialize_side` copies them beside the excerpt at their repository-relative paths so the entry-point mapper sees the same layout the project has. `src/openultrasast/mapping.py` gains `_route_manifest_entry_points`, which reads `operationId` and the operation's `security` block out of a YAML or JSON document and attributes the record to the module the operation id points at. Trap pairs whose fixed side is a different, correctly guarded handler carry `fix_function` so the fixed side gets *its* registration rather than the vulnerable side's.
Decisions: unscorable pairs short-circuit before any run; K runs share one classification; verifier outcomes never change detection or silence, they change the reported rung.
### One evolve round
```mermaid
flowchart TB
Start[failure facts for target family] --> Propose[proposer drafts one structural change]
Propose --> Refuse{touches holdout or exceeds length cap}
Refuse -->|yes| Reject[reject and record]
Refuse -->|no| Mini[train minibatch of target family]
Mini --> MiniOk{no decrease}
MiniOk -->|no| Reject
MiniOk -->|yes| Hold[target family full holdout]
Hold --> HoldOk{train and holdout non negative and one strictly positive}
HoldOk -->|no| Reject
HoldOk -->|yes| Sweep[every other family holdout]
Sweep --> SweepOk{negative flips within each budget}
Decisions: the proposer never sees holdout inputs, outputs or scores; cost is metered per stage and the round is reverted when the cap is crossed at any stage; ties reject.
| Scorer | logic | class-aware pair outcomes and statistics | 3.x, 4.x | Families (P0); findings from Detectors and rungs from Verifiers are inputs, not imports (P1) | Service |
- Validates every loaded catalog `family` label against the taxonomy at classification time and rejects a fabricated id by name; the corpus loader stores the label as an opaque string.
- Postcondition: every reviewed pair whose label disagrees with the classification appears once in the review queue with both answers and the tier.
#### Scorer
| Field | Detail |
|---|---|
| Intent | Score one pair per family from detector findings, aggregate per family and slice, decide reliable change |
| Requirements | 3.1–3.6, 4.1, 4.5 |
**Responsibilities & Constraints**
- Detection: a finding whose `family:` tag relates `same` or `parent_child` to the labeled family and whose line lies in the labeled function; `fabricated` scores the maximum penalty; text is never inspected.
- Silence: no `same`/`parent_child` finding of the labeled family inside the labeled function on the fixed side; other findings are counted in `other_findings`, never as leaks.
- K runs per pair at every evaluation (baseline K = 5, rounds K ≥ 3 as 3.5 requires); reliable change against the last blessed round via a paired test on per-pair outcomes; negative flips per family.
-`export_mechanisms`, `evaluate_loo`, `evaluate_profiles`, `evaluate_mechanism_profiles`, `propose_mechanism_edits` and `build_pair_signals` obtain teachers only through `teachers`; refusals are recorded as degradations `{"stage": "learning", "reason": "holdout_pair_refused", "pairs": [...]}`.
#### Detectors
| Field | Detail |
|---|---|
| Intent | Load frozen per-family configurations and run the family detector with whole-file context and curated tools |
| Requirements | 6.1, 6.2, 6.3, 6.4, 6.5 |
**Responsibilities & Constraints**
- A configuration is a directory `learning/configs/<family>/` with `prompt.md` (length cap enforced at load), `checklist.md`, `tools.toml` (subset of `read_file`, `grep_repo`, `find_refs`, `read_definition`, `entry_points`, `flows`, `obligations`), `budget.toml` (`max_steps`, `max_cost_usd`, `max_chars`), `hard_negatives.jsonl`, `counterexamples.jsonl`, `version`.
- Round zero writes every family's directory from the current hunter prompt; a round rewrites only its target family and bumps that `version`.
- The runner builds the user message from the whole file plus resolved definitions, runs `run_tool_hunter` with the family's tools and prompt, and tags findings `family:<id>`, `detector:<id>@<version>`; the generalist is the `unknown` family's configuration.
**Contracts**: Service [x] State [x]
```python
@dataclass(frozen=True)
classFamilyConfig:
family:str
version:str
prompt:str
checklist:str
tools:tuple[str,...]
max_steps:int
max_cost_usd:float
max_chars:int
hard_negatives:tuple[dict[str,object],...]
counterexamples:tuple[dict[str,object],...]
defload_family_configs(configs_dir:Path,taxonomy:FamilyTaxonomy)->dict[str,FamilyConfig]# raises ConfigError on cap or tool violations
- Adapter: `thinking` disabled and `temperature 0` for detectors and classifier; `reasoning_content` replayed; `response_format json_object` with one retry on empty content; `logprobs` on request; usage fields normalized (`prompt_cache_hit_tokens`) into cost accounting.
| Intent | Confirm a claim through the family's oracle, or refuse to raise it |
| Requirements | 7.1–7.6 |
**Responsibilities & Constraints**
-`CanaryVerifier` (injection, path, deserialization, and command execution inside injection): writes a snippet from a family template that imports the target function, plants a canary (row in an in-memory database, file under `/scratch`, object with a marker method) and prints `CANARY:<token>` only when the vulnerability fires; the oracle is the token in stdout, never the exit code. The snippet passes `check_snippet_safety` unchanged.
-`StaticVerifier` (access control): the obligations checker's finding on the same function raises to `static_corroboration`.
-`second_judge`: a `deepseek-v4.1-flash` call with the snippet, the oracle output and the claim, answering `json_object``{"confirms": bool, "reason": str}`; a proven claim is published only when it confirms.
-`run_baseline`: clone the current hunter prompt into every family directory, run K = 5 over scorable vendored pairs of the selected slices, compute per-family negative-flip rate between runs as `NoiseFloor`, persist `noise-floors.json`, publish round zero. Artifacts and floors are keyed by detector model (`baseline/<model>/`), so a second model is baselined by changing `[models].hunter` and re-running; `publish` renders a model comparison table from every baseline present. This is the maintainer's reconsideration path if the default model underperforms; configurations are never touched by a model swap.
- Idempotency: a round directory is written under `.openultrasast/learning/rounds/<n>/` with the snapshot, proposal, scores and trajectories; rerunning a round number refuses.
- Every stage scores with K runs as the scorer requires; the noise floor is the tolerance for the cross-family sweep, not a substitute for repeated runs.
classHarnessXProposer:...# MetaAgent.evolve with allowed_write_roots = the family directory only; degrades to None with a recorded reason without the extra
```
#### Publish
| Field | Detail |
|---|---|
| Intent | Regenerate every published number from one command and commit its artifacts |
- Writes `<date>-<slice>-<family>.json` artifacts, a markdown table with scorable/unscorable counts, taxonomy version, noise floor and cost per point beside every number, and replaces the roadmap's measured section between markers.
## Data Models
### Domain Model
- Aggregates: `FamilyTaxonomy` (versioned, immutable per version); `FamilyConfig` (per family, versioned, frozen between rounds); `RoundRecord` (append-only); `PairCase` (owned by the corpus, gains two additive fields).
- Invariants: a finding carries exactly one `family:` tag; a pair has at most one `unscorable` reason; a round targets exactly one family; the archive maps a pair to one winning configuration per family.
- Catalog row additions: `[[pair.expected]] family = "<id>"` (opaque at load, validated by the Classifier), `known_limit = "identical_twin"` for the five twins, `fix_date = "YYYY-MM-DD"` populated by the catalog generator from recipe metadata where available (rows without it are counted as undated in the post-cutoff slice); `unscorable` is computed, never stored.
- Journal and archive JSONL as above; `noise-floors.json`; `review-queue.jsonl` entries `{pair, label_family, classified, tier, confidence}`.
## Error Handling
- Taxonomy or configuration deviations fail fast at load with the field named (`FamiliesError`, `ConfigError`).
- Missing extras degrade with a recorded reason: no HarnessX → `ScriptedProposer` or `proposer_unavailable`; no TypeScript grammar → rows `unsupported_language`; no sandbox → `NoVerifier` for canary families with `sandbox_unavailable`.
- Endpoint errors: transient retries as today; empty `json_object` content retried once then recorded; a round whose endpoint fails mid-stage is reverted with `endpoint_failed`.
- Holdout refusal, cost overrun and safety rejection are outcomes, not exceptions, and always land in the journal.
- Monitoring: every stage runs under `HarnessRuntime.run_stage`; cost per stage and per round is written to the round directory.
## Testing Strategy
- Unit: taxonomy loader rejects an eleventh family and a missing verifier kind (1.1, 1.2); `related` returns `parent_child` for a CWE child and `fabricated` for an unknown id (3.1); classifier tier 1 decides from a rule id without a client and tier 2 falls to `unknown` under the threshold with a scripted client (2.1, 2.2); scorer counts detection only inside the labeled function and never from text (3.1, 3.6); silence ignores other-family findings (3.2); directional bias and negative flips on synthetic run sets (3.3, 3.5); `unscorable_reason` on an identical twin and an unsupported language (4.1); `teachers` excludes holdout and `refuse_if_holdout` names the pair (5.1, 5.2); configuration loader rejects a prompt over the cap and a tool outside the allowed set (6.1, 6.5); endpoint resolution prefers DeepSeek over OpenRouter and never touches the embedding variables (6.6); canary snippet passes safety and the oracle keys on the token (7.2); acceptance rule rejects ties and budget breaches (9.5); journal round-trips a record and the rejected buffer (9.3, 9.6).
- Integration: `pairs --hunter` with a scripted client on the shared handler fixture yields family outcomes and the manifest block (3.x, 6.3); `export_mechanisms` and `evaluate_loo` teach from train only and record refusals (5.1, 5.3); `run_baseline` with a scripted client writes noise floors and round-zero artifacts (8.x); `run_learning_round` with `ScriptedProposer` accepts a helpful change, rejects a tie, reverts on cost and journals attribution (9.x); `run_regression(oracle=)` returns `TRIGGERABLE` on the token with `FakeSandboxRunner` (7.2).
- Corpus: TypeScript rows parse under the extra (10.1); the five twins load as `known_limit:identical_twin` and are excluded from denominators (10.3, 4.1); `test_fixture_difficulty` fails on a fixture easier than its named rows (10.4); post-cutoff slice reported separately (10.5).
- Gates: `gate`, `map_gate`, `pair_gate` stdout byte-identical to the pre-feature commit; extra-free suite green with semantic and harnessx tests skipping (11.5).
- Live (maintainer, not CI): round zero on vibe-py and agent-vfc web families through DeepSeek, artifacts committed under `benchmarks/measurements/`.
## Security Considerations
- Every prompt and every persisted trajectory passes `redact_secrets`; verifier templates, oracle code and gold labels are outside the proposer's write roots; snippets keep today's safety bans; the sandbox stays network-off and workspace read-only; the second judge sees the snippet and oracle output, never the labels.
## Performance & Cost
- Round zero: K = 5 over ~250 vendored pairs, both sides, whole-file context ≈ 25M input tokens; at DeepSeek flash peak pricing under $15, off-peak half. A round: minibatch + holdout + sweep ≈ one baseline pass; `round_cost_cap_usd` default 10. Concurrency bounded by the runtime's existing budget controls.
- Phase 2 (live): round zero on web families; noise floors; classifier tier 2 measured; review queue worked.
- Phase 3: verifiers for the four canary families and the static verifier; second judge.
- Phase 4: evolve rounds with `ScriptedProposer` first, then `HarnessXProposer`; obligations lever as the access-control family's first round.
- Rollback: every round is a directory snapshot; `publish` regenerates from artifacts, so a bad round is reverted by restoring the family directory and re-publishing.
