mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 12:39:28 +00:00
docs: plain-English pass over the active specs
Ran the humanizer rules over 79 prose files in the active specs (brief, design, requirements, research and narrative docs). Removed em dashes, significance inflation, forced rule-of-three, bold-label list items and filler. Skipped tasks.md checklists and the retired specs (harnessx-*, openrouter-sast-harness, unseen-repo-evaluation). Every number, path, command, hash and identifier was preserved: checked each file against HEAD for drift in numbers, inline code and code fences, zero drift across all 79 files. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
b08f372022
commit
39a1873e5b
@@ -62,8 +62,8 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
|
||||
|
||||
## Scope
|
||||
|
||||
- **In**: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
|
||||
- **Out**: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
- In: items 1–8; roadmap rewrite from one command; a closure decision on `authorization-obligations` 4.3, 5 and 6.
|
||||
- Out: new static shape families; hand-written facts beyond what a round proposes and validates; changing the evidence ladder or dispositions; any merge gate on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
@@ -80,14 +80,14 @@ Principle for the roadmap: **classify first, verify before reporting, improve on
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- **Upstream**: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
|
||||
- **Downstream**: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
|
||||
- Upstream: `pair-corpus-honesty` (labels, tiers, splits, `known_limit`), `corpus-seeded-mechanisms` (exporter, LOO, lever), `tool_hunter` and `provider/openrouter`, `harness_ext`, HarnessX 0.1.0 (`meta_harness`, `rl`, `tracing.journal`), `tree-sitter-overlay-extra`, the sandbox and regress recipes.
|
||||
- Downstream: every roadmap number; `harnessx-self-improving-rulesets` (its evolve step is this spec); `authorization-obligations` 4.3/5/6; scan reports (findings carry class, detector and verifier).
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
|
||||
- **Extends**: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
|
||||
- **Folds in**: the retired `closed-loop-integrity` brief.
|
||||
- **Re-homes**: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
|
||||
- Extends: `pair-corpus-honesty` Req 1 (scorer) and Req 9/10 (row classification); `corpus-seeded-mechanisms` Req 4/5 (split by construction); `three-stage-scan` MAP (per-class hunters as proposers); `propose-adjudicate-prove` (verifiers keyed by class); `harnessx-self-improving-rulesets` (the evolve loop lands here).
|
||||
- Folds in: the retired `closed-loop-integrity` brief.
|
||||
- Re-homes: `authorization-obligations` 4.3 and the corpus half of 6.1; holds 5.
|
||||
|
||||
## Constraints
|
||||
|
||||
|
||||
@@ -150,20 +150,20 @@ tests/test_learning_families.py, test_learning_classify.py, test_learning_scorin
|
||||
```
|
||||
|
||||
### Modified Files
|
||||
- `src/openultrasast/pairs.py` — `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
|
||||
- `src/openultrasast/benchmark.py` — `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
|
||||
- `src/openultrasast/tool_hunter.py` — `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
|
||||
- `src/openultrasast/hunter_tools.py` — four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
|
||||
- `src/openultrasast/provider/openrouter.py` — `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
|
||||
- `src/openultrasast/config.py` — `[models]` gains `chat_base_url`, `chat_api_key_env`, `judge`; `[learning]` section (`families_path`, `configs_dir`, `k_runs = 5`, `round_cost_cap_usd`, `minibatch`, `cutoff_date`); `load_dotenv` unchanged, tests inject endpoints explicitly.
|
||||
- `src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py` — teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
|
||||
- `src/openultrasast/regress/candidate.py`, `regress/verdict.py` — `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
|
||||
- `src/openultrasast/cli.py` — `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
|
||||
- `pyproject.toml` — `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
|
||||
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml` — the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
|
||||
- `src/openultrasast/pairs.py`: `PairCase.unscorable: str | None` computed at load (identical twin via normalized body hash, `known_limit` passthrough); duplicate report across pairs before split; `split_by_repository` helper honoring existing `split` fields and reporting straddles; `_evaluate_hunter_pair` delegates matching and leaks to `learning.scoring` through an injected callable so `pairs` does not import `learning`; `build_pair_signals` gains a `split` filter.
|
||||
- `src/openultrasast/benchmark.py`: `ExpectedFinding.family: str | None = None` (additive, stored as an opaque string; the Classifier validates loaded labels against the taxonomy so `pairs` and `benchmark` never import `learning`).
|
||||
- `src/openultrasast/tool_hunter.py`: `run_tool_hunter` gains `system_prompt`, `context_files`, `tools`, `max_chars` and `tags` parameters with today's values as defaults; finding parsing accepts an optional `family` field; the reply loop replays `reasoning_content`.
|
||||
- `src/openultrasast/hunter_tools.py`: four new tools beside `read_file`, `grep_repo`, `find_refs`: `read_definition(symbol)`, `entry_points(path)`, `flows(path, function)`, `obligations(path, function)`, each clamped to the repository and returning identifiers and line spans only; `HUNTER_TOOLS` schemas extended.
|
||||
- `src/openultrasast/provider/openrouter.py`: `OpenRouterChatClient` gains `extra_body` (thinking, response_format, logprobs) and retains the `reasoning_content` field on assistant messages; replaying it across tool turns is the hunter loop's job (`tool_hunter`), not the client's. No behavior change for existing callers.
|
||||
- `src/openultrasast/config.py`: `[models]` gains `chat_base_url`, `chat_api_key_env`, `judge`; `[learning]` section (`families_path`, `configs_dir`, `k_runs = 5`, `round_cost_cap_usd`, `minibatch`, `cutoff_date`); `load_dotenv` unchanged, tests inject endpoints explicitly.
|
||||
- `src/openultrasast/semantic/seed.py`, `semantic/loo.py`, `improve/evolve.py`: teacher selection through `learning.split.teachers`; refusals recorded; `evaluate_loo` teaches from train only.
|
||||
- `src/openultrasast/regress/candidate.py`, `regress/verdict.py`: `run_regression` accepts an optional `oracle: Callable[[SandboxResult], bool]`; `verdict_from_result` uses the oracle when given, exit code otherwise.
|
||||
- `src/openultrasast/cli.py`: `ousast learning classify|score|baseline|round|publish`; `pairs --hunter` uses the family scorer and `--k-runs`; `[models].hunter` resolution through `learning.endpoint`.
|
||||
- `pyproject.toml`: `tree-sitter-typescript>=0.23` in the semantic extra; `src/openultrasast/semantic/extra.py` maps `typescript`/`tsx`.
|
||||
- `benchmarks/pairs/vibe-py/catalog.toml` and `recipes.toml`: the five identical twins gain `known_limit = "identical_twin"`; absence rows gain `family` and `mode = "handler_context"` (re-harvest is a maintainer step).
|
||||
- **Cross-file registration context (amended during task 4.1).** Some frameworks keep the route table out of the handler module: connexion registers by `operationId` in an OpenAPI document, an Express application registers in a router file. A single-file excerpt of such a handler can never contain its registration, so Req 10.2 is unreachable for those rows without a second file. A recipe may therefore name `context = ["<repo-relative path>"]`; `benchmarks/pairs/harvest.py` fetches each document at the same two commits, extracts only the stanza that names the handler (`extract_registration_context`), and writes it under `<name>/context/<side>/<relpath>` with its own provenance header and the same redaction. The catalog carries `[[pair.context]]` rows, `PairCase.context_files` holds `(relpath, vuln, fixed)`, and `_materialize_side` copies them beside the excerpt at their repository-relative paths so the entry-point mapper sees the same layout the project has. `src/openultrasast/mapping.py` gains `_route_manifest_entry_points`, which reads `operationId` and the operation's `security` block out of a YAML or JSON document and attributes the record to the module the operation id points at. Trap pairs whose fixed side is a different, correctly guarded handler carry `fix_function` so the fixed side gets *its* registration rather than the vulnerable side's.
|
||||
- `benchmarks/pairs/mechanisms.toml` — unchanged; families reference mechanism ids.
|
||||
- `.kiro/steering/roadmap.md` — rewritten by `publish` from artifacts.
|
||||
- `benchmarks/pairs/mechanisms.toml`: unchanged; families reference mechanism ids.
|
||||
- `.kiro/steering/roadmap.md`: rewritten by `publish` from artifacts.
|
||||
|
||||
## System Flows
|
||||
|
||||
|
||||
@@ -11,9 +11,9 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: the class taxonomy and its classifier; the class-aware pair scorer and unscorable-row accounting; split enforcement for every learning path; per-family detector configurations and their routing; class-keyed verifiers and reporting rungs; the round-based evolve loop with attribution; the corpus repairs the loop needs; publishing every number with its denominator and noise floor.
|
||||
- **Out of scope**: finding bugs in JavaScript interpreters, WebGL renderers or browser engines; memory-safety fuzzing or crash oracles for C and C++ (the vfc slice stays a measured, non-gating slice under one coarse family); new hand-written static shape families; changes to the evidence ladder or dispositions; merge gates on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
- **Adjacent expectations**: the pair corpus (`pair-corpus-honesty`) keeps owning labels, tiers, splits and `known_limit`; `corpus-seeded-mechanisms` keeps its exporter, leave-one-out and lever, which this feature subordinates to the split and the class attribution; `authorization-obligations` keeps its checker, whose corpus tasks 4.3 and 6.1 move here and whose lever task 5 becomes the access-control family's first round; the local detection, map and pair gates stay byte-identical; HarnessX and tree-sitter grammars remain optional extras and every step that needs them degrades to a recorded reason without them.
|
||||
- In scope: the class taxonomy and its classifier; the class-aware pair scorer and unscorable-row accounting; split enforcement for every learning path; per-family detector configurations and their routing; class-keyed verifiers and reporting rungs; the round-based evolve loop with attribution; the corpus repairs the loop needs; publishing every number with its denominator and noise floor.
|
||||
- Out of scope: finding bugs in JavaScript interpreters, WebGL renderers or browser engines; memory-safety fuzzing or crash oracles for C and C++ (the vfc slice stays a measured, non-gating slice under one coarse family); new hand-written static shape families; changes to the evidence ladder or dispositions; merge gates on non-local slices; vendoring new pairs beyond re-harvesting existing recipes; fine-tuning model weights.
|
||||
- Adjacent expectations: the pair corpus (`pair-corpus-honesty`) keeps owning labels, tiers, splits and `known_limit`; `corpus-seeded-mechanisms` keeps its exporter, leave-one-out and lever, which this feature subordinates to the split and the class attribution; `authorization-obligations` keeps its checker, whose corpus tasks 4.3 and 6.1 move here and whose lever task 5 becomes the access-control family's first round; the local detection, map and pair gates stay byte-identical; HarnessX and tree-sitter grammars remain optional extras and every step that needs them degrades to a recorded reason without them.
|
||||
|
||||
## Requirements
|
||||
|
||||
@@ -87,7 +87,7 @@ OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The
|
||||
1. The Learning Harness shall hold one detector configuration per family consisting of a prompt under a stated length cap, a curated tool set, a step budget, a cost budget, an evidence checklist and a set of hard negatives and counterexamples.
|
||||
2. While a round evolves one family, the Learning Harness shall keep every other family's configuration unchanged.
|
||||
3. When a code region is classified, the Learning Harness shall run the detector of every admitted family plus a generalist detector, and shall record on each finding the family, the detector configuration version and the verifier outcome.
|
||||
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve, not only an excerpt around a hotspot.
|
||||
4. The Learning Harness shall give a detector the whole file of a region and the files its curated tools resolve instead of limiting input to an excerpt around a hotspot.
|
||||
5. The Learning Harness shall expose to a detector only curated tools that read, search and resolve symbols within the scanned repository, and no general shell.
|
||||
6. The Learning Harness shall run detectors, the classifier's model tier and the evolve proposer on a configurable chat endpoint that is independent of the embedding endpoint, with a cheap model as the default and a stronger model reserved for the second judge and for review-queue disagreements.
|
||||
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
## Summary
|
||||
|
||||
- **Feature**: `learning-harness`
|
||||
- **Discovery Scope**: Complex Integration (extension of pairs, tool hunter, provider, regress, improve; adoption of HarnessX meta-harness; corpus repairs)
|
||||
- **Key Findings**:
|
||||
- Feature: `learning-harness`
|
||||
- Discovery Scope: Complex Integration (extension of pairs, tool hunter, provider, regress, improve; adoption of HarnessX meta-harness; corpus repairs)
|
||||
- Key Findings:
|
||||
- The tool hunter already beats the overlay 12/14 vs 5/14 on the class-aware, scorable vibe-py holdout; the published 0/17 was the text-token scorer.
|
||||
- Every learning path ignores the split (`evaluate_profiles`, `evaluate_mechanism_profiles`, `propose_mechanism_edits`, `evaluate_loo`, `build_pair_signals`, `export_mechanisms`); one teacher rule fixes all of them.
|
||||
- The sandbox verdict is exit-code only and snippets are compile-only templates; a canary oracle is an additive callback that respects the existing safety bans.
|
||||
@@ -15,13 +15,13 @@
|
||||
## Research Log
|
||||
|
||||
### Codebase seams (2026-09-06, Explore subagent over `src/openultrasast`)
|
||||
- **Context**: design an extension without re-reading every module.
|
||||
- **Findings**: `run_tool_hunter(root, hotspots, *, client, model, max_steps)` with three curated tools and `suspicion` findings tagged `tool-hunter`; `ChatClient.complete(*, model, messages, tools, timeout_seconds)`; `OpenRouterChatClient.complete_chat` always sends `temperature: 0` and drops `reasoning_content`; `_hunter_matches_expected` is text-token based and every fixed-side finding leaks under `fix_policy = silent`; `run_regression(..., sandbox, sandbox_limits, images)` with `verdict_from_result` on exit code; `check_snippet_safety` bans sockets, curl and workspace writes; `MechanismStore` append-only JSONL with tombstones; `improve.run_round` journal has no hypothesis, attribution or cost fields and its regression check is per provenance profile; `load_dotenv` no-ops under pytest; `pyproject` semantic extra lacks TypeScript; `pair_gate` scores only vendored `local` pairs.
|
||||
- **Implications**: the design adds parameters and additive fields instead of replacing modules; the family scorer is injected into the hunter path so `pairs` never imports `learning`; the oracle is an optional callback on `run_regression`; endpoint resolution takes an explicit override for tests.
|
||||
- Context: design an extension without re-reading every module.
|
||||
- Findings: `run_tool_hunter(root, hotspots, *, client, model, max_steps)` with three curated tools and `suspicion` findings tagged `tool-hunter`; `ChatClient.complete(*, model, messages, tools, timeout_seconds)`; `OpenRouterChatClient.complete_chat` always sends `temperature: 0` and drops `reasoning_content`; `_hunter_matches_expected` is text-token based and every fixed-side finding leaks under `fix_policy = silent`; `run_regression(..., sandbox, sandbox_limits, images)` with `verdict_from_result` on exit code; `check_snippet_safety` bans sockets, curl and workspace writes; `MechanismStore` append-only JSONL with tombstones; `improve.run_round` journal has no hypothesis, attribution or cost fields and its regression check is per provenance profile; `load_dotenv` no-ops under pytest; `pyproject` semantic extra lacks TypeScript; `pair_gate` scores only vendored `local` pairs.
|
||||
- Implications: the design adds parameters and additive fields instead of replacing modules; the family scorer is injected into the hunter path so `pairs` never imports `learning`; the oracle is an optional callback on `run_regression`; endpoint resolution takes an explicit override for tests.
|
||||
|
||||
### DeepSeek platform API (2026-09-06, api-docs.deepseek.com)
|
||||
- **Findings**: base `https://api.deepseek.com` (no `/v1` in current docs), Anthropic-shape endpoint at `/anthropic`; models `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`; 1M context, 384K output; thinking on by default (`thinking: {"type": "disabled"}` to turn off; `temperature` ignored while thinking); tools with `tool_choice`, `reasoning_content` must be replayed on tool turns; `response_format json_object` only, occasional empty content; `logprobs`/`top_logprobs` documented, unverified in thinking mode; no `seed`, `n`, `logit_bias`; flash 0.44/1.32 USD per M in/out peak, cache hit 0.014, pro 1.32/3.96, off-peak half; concurrency 2500 flash / 500 pro; no embeddings endpoint.
|
||||
- **Implications**: detectors and classifier run with thinking disabled and temperature 0; determinism is best-effort and the K-run noise floor absorbs it; the classifier's model tier uses `logprobs` averaged and falls to `unknown`; embeddings stay on OpenRouter; cost accounting reads DeepSeek's cache-hit usage fields.
|
||||
- Findings: base `https://api.deepseek.com` (no `/v1` in current docs), Anthropic-shape endpoint at `/anthropic`; models `deepseek-v4-flash`, `deepseek-v4-pro`, `deepseek-v4-flash-vision-exp`; 1M context, 384K output; thinking on by default (`thinking: {"type": "disabled"}` to turn off; `temperature` ignored while thinking); tools with `tool_choice`, `reasoning_content` must be replayed on tool turns; `response_format json_object` only, occasional empty content; `logprobs`/`top_logprobs` documented, unverified in thinking mode; no `seed`, `n`, `logit_bias`; flash 0.44/1.32 USD per M in/out peak, cache hit 0.014, pro 1.32/3.96, off-peak half; concurrency 2500 flash / 500 pro; no embeddings endpoint.
|
||||
- Implications: detectors and classifier run with thinking disabled and temperature 0; determinism is best-effort and the K-run noise floor absorbs it; the classifier's model tier uses `logprobs` averaged and falls to `unknown`; embeddings stay on OpenRouter; cost accounting reads DeepSeek's cache-hit usage fields.
|
||||
|
||||
### Measurement that triggered the spec (2026-09-06)
|
||||
- See `brief.md` Problem section and `benchmarks/measurements/2026-09-06-*`.
|
||||
@@ -38,37 +38,37 @@
|
||||
## Design Decisions
|
||||
|
||||
### Decision: families are the only routing and scoring key
|
||||
- **Context**: 1.4, 3.6; text tokens and CWE ids were the proxy.
|
||||
- **Alternatives**: fine CWE routing; mechanism ids as key.
|
||||
- **Selected**: ten coarse families with CWE and mechanism as attributes; hierarchical relation for partial credit.
|
||||
- **Rationale**: fine CWE from code is unsolved (≤14.7% top-1) and coarse families are what worked in AIxCC and MoEVD.
|
||||
- **Trade-offs**: less granular reporting; compensated by attributes on findings.
|
||||
- Context: 1.4, 3.6; text tokens and CWE ids were the proxy.
|
||||
- Alternatives: fine CWE routing; mechanism ids as key.
|
||||
- Selected: ten coarse families with CWE and mechanism as attributes; hierarchical relation for partial credit.
|
||||
- Rationale: fine CWE from code is unsolved (≤14.7% top-1) and coarse families are what worked in AIxCC and MoEVD.
|
||||
- Trade-offs: less granular reporting; compensated by attributes on findings.
|
||||
|
||||
### Decision: the scorer is injected into the hunter path, not imported by `pairs`
|
||||
- **Context**: dependency direction; `pairs` is owned by `pair-corpus-honesty`.
|
||||
- **Selected**: `_evaluate_hunter_pair` accepts a matcher/leak callable; the CLI wires `learning.scoring`.
|
||||
- **Trade-offs**: one more parameter; keeps `pairs` free of `learning` and the overlay path unchanged.
|
||||
- Context: dependency direction; `pairs` is owned by `pair-corpus-honesty`.
|
||||
- Selected: `_evaluate_hunter_pair` accepts a matcher/leak callable; the CLI wires `learning.scoring`.
|
||||
- Trade-offs: one more parameter; keeps `pairs` free of `learning` and the overlay path unchanged.
|
||||
|
||||
### Decision: canary oracle as an optional callback on `run_regression`
|
||||
- **Context**: 7.2; exit code cannot express "the vulnerability fired".
|
||||
- **Alternatives**: a parallel verifier runner; parsing stderr.
|
||||
- **Selected**: `oracle: Callable[[SandboxResult], bool] | None`; token in stdout.
|
||||
- **Trade-offs**: minimal change to regress; templates per family live in `learning/verifiers/`.
|
||||
- Context: 7.2; exit code cannot express "the vulnerability fired".
|
||||
- Alternatives: a parallel verifier runner; parsing stderr.
|
||||
- Selected: `oracle: Callable[[SandboxResult], bool] | None`; token in stdout.
|
||||
- Trade-offs: minimal change to regress; templates per family live in `learning/verifiers/`.
|
||||
|
||||
### Decision: HarnessX proposer behind the seam, scripted proposer first
|
||||
- **Context**: 9.x with core `dependencies = []`.
|
||||
- **Selected**: `Proposer` protocol; `HarnessXProposer` composes `MetaAgent.evolve` with `allowed_write_roots` = the family directory; `ScriptedProposer` for tests and phase 4 start.
|
||||
- **Trade-offs**: two proposers; the loop is testable offline and the extra stays optional.
|
||||
- Context: 9.x with core `dependencies = []`.
|
||||
- Selected: `Proposer` protocol; `HarnessXProposer` composes `MetaAgent.evolve` with `allowed_write_roots` = the family directory; `ScriptedProposer` for tests and phase 4 start.
|
||||
- Trade-offs: two proposers; the loop is testable offline and the extra stays optional.
|
||||
|
||||
### Decision: noise floor from K = 5 baseline runs is the regression budget
|
||||
- **Context**: 8.2, 9.5; no public numeric negative-flip budget exists.
|
||||
- **Selected**: per-family negative-flip rate between seed runs; `budget_flips = ceil(rate × scorable)`.
|
||||
- **Trade-offs**: a noisy family gets a loose budget; reported beside the number so it is visible.
|
||||
- Context: 8.2, 9.5; no public numeric negative-flip budget exists.
|
||||
- Selected: per-family negative-flip rate between seed runs; `budget_flips = ceil(rate × scorable)`.
|
||||
- Trade-offs: a noisy family gets a loose budget; reported beside the number so it is visible.
|
||||
|
||||
### Decision: DeepSeek adapter wraps the existing OpenRouter client
|
||||
- **Context**: 6.6; both clients share one base URL today.
|
||||
- **Selected**: `ChatEndpoint` resolution order DeepSeek → OpenRouter → scripted; adapter adds `extra_body` and `reasoning_content` replay; embeddings untouched.
|
||||
- **Trade-offs**: one adapter class; no new dependency.
|
||||
- Context: 6.6; both clients share one base URL today.
|
||||
- Selected: `ChatEndpoint` resolution order DeepSeek → OpenRouter → scripted; adapter adds `extra_body` and `reasoning_content` replay; embeddings untouched.
|
||||
- Trade-offs: one adapter class; no new dependency.
|
||||
|
||||
## Risks & Mitigations
|
||||
- Router error routes a region to the wrong family → multi-label routing plus the generalist; router measured separately against random and oracle.
|
||||
|
||||
Reference in New Issue
Block a user