diff --git a/.kiro/specs/harnessx-removal/design.md b/.kiro/specs/harnessx-removal/design.md new file mode 100644 index 0000000..21ce4a3 --- /dev/null +++ b/.kiro/specs/harnessx-removal/design.md @@ -0,0 +1,473 @@ +# Design Document + +## Overview + +HarnessX leaves the product in one reviewable sequence, and the plane's memory layer takes over what HarnessX was +meant to drive. Three facts from the code shape the design: + +- **No HarnessX capability is measured, and one of them never existed.** The LLM paths are reached only through + config keys, not CLI flags (`scan` has `path`, `--mode`, `--config`, `--fail-on`; `cli.py:163-167`), and + `benchmarks/` holds no run of any of them (no `hx-hunter:` finding, no `llm-panels` decision). The + `MetaAgent.evolve` proposer is only a docstring (`improve/evolve.py:8-9`, `harness_ext.py:7-8`). No code calls it. +- **The plane already does the LLM work better, and outside `ousast scan`.** `verify` (two passes, known callers + from `repo-facts`) plus `agree` (2-of-3 with the tie-break) met the ai-service-plane gate + (`benchmarks/independent/plane-increment-2.json`: 16/20 declared sites agreed, $0.0205 per candidate). The + standard-mode LLM hunter that users actually reach, the tool hunter at the MAP stage (`cli.py:631-640`, + `tool_hunter.resolve_hunter_client`), is not HarnessX and stays. +- **Everything deterministic already runs without the extra.** Every guarded branch falls back to a deterministic + function (`run_hunter_pool`, `verify_findings`, `fuse_finding`). Removal therefore makes the fallback the only + path. Quick mode, `benchmark`, `pairs`, `improve` and the gates never enter a guarded branch. + +So every HarnessX capability in `ousast scan` is **retired** with its deterministic path kept. The plane's `verify` +and `agree` tasks are the documented agentic path (`ousast plane run `). The proposer is **carried by the +memory layer**: a persistent store, keyed by repository and pin, that plane Runs write through a model-free +`remember` task. From that store a deterministic generator emits `RuleStatusEdit` proposals into the unchanged +`EvolveValidator` and gate. The improvement loop becomes a plane Run whose only Model-bound tasks are the verify +passes. + +## Architecture + +``` +Before After +ousast scan --mode standard ousast scan --mode standard + hunter_pool: HxScanOrchestrator | run_hunter_pool hunter_pool: run_hunter_pool (only path) + verify: verify_judge (llm-judge) | verify_findings verify: verification.verify_findings + fusion: llm panels | deterministic panels fusion: deterministic panels (fuse_findings) + MAP: tool hunter ([models] hunter, DeepSeek) MAP: tool hunter (unchanged) +harness_ext.py, hunter_harness.py, verify_judge.py deleted +stage_processors.host_under_harnessx deleted; deterministic pipeline kept +[harnessx] config, extra, mypy override, lock one load warning; extra/override/lock gone +improve: benchmark per-rule proposer (+ MetaAgent improve: benchmark proposer + memory proposer (--memory) + "plugs in" -- never implemented) + plane Run (per case) + facts -> va, vb -> agree -> vc -> final (existing) + alerts (new, no model) ------------------+ + remember (new, no model) <- facts, final, alerts, va/vb/vc + plane Run (loop tail, once) + measure -> propose -> improve(validate+gate) (no model) +~/ousast-results/plane/memory/ the store: facts by content, rows by repo + pin +``` + +New code: `src/openultrasast/plane/memory.py` (store, ingest, seed; host side), +`src/openultrasast/plane/tasks/{alerts,remember,measure,propose,improve}.py`, +`src/openultrasast/improve/memory.py` (deterministic proposal rules). None of it is in the reconciler. The +reconciler gains only `mark_done` (below). It is 465 lines today, and `tests/test_plane_reconciler.py:759-761` +keeps it under 500. + +## Components and Interfaces + +### 1. Fate of each HarnessX capability (Requirement 1) + +| Capability | Where today | Fate | What the user loses | +| --- | --- | --- | --- | +| LLM hunter pool (`HxScanOrchestrator`) | `hunter_harness.py:49-185`; gated in `cli.py:400-456` (`hx_hunter` at :405, orchestrator at :438-448, degradation at :452-455) | **Retired.** `run_hunter_pool` becomes the only standard-mode hunter pool. LLM hunting of a repository stays available in two places: the MAP-stage tool hunter in `ousast scan --mode standard/deep` with `[models] hunter` (`cli.py:631-640`, not HarnessX), and plane `verify` via `ousast plane run plane/runs/validation-46.yaml` | `hx-hunter::` findings from an anthropic/openai/litellm agent (`hunter_harness.py:155`). No run of this path is recorded under `benchmarks/`, so no measured TP is lost. | +| LLM judge | `verify_judge.py:22-63`; gated in `cli.py:406, 499-511` | **Retired.** `verification.verify_findings` is called directly and `verify_judge.py` is deleted (it is only the dispatcher and the judge). The plane's LLM verification is `verify` + `agree` (+ `vc`/`final` tie-break) | Per-finding `plausible/unsupported` verdicts that set `ACCEPTED/REJECTED` (`verify_judge.py:83-88`). Unmeasured. `[models] judge` (model-grounded-detection, `model/endpoint.py:195`) is a different capability and stays. | +| LLM fusion panels | `fusion.py:194-326` (`_fuse_with_llm_panels`, `_run_llm_panels`); gated in `cli.py:519-541` | **Retired.** The deterministic panels (`_panel_a`, `_panel_b`, `_reconcile`, `fusion.py:101-167`) become the only path. `fuse_findings_dispatch` is renamed `fuse_findings(findings, verifications, *, high_assurance, mitigated_ids)`. The plane counterpart of independent cross-examination is `agree` across independent passes with the 2-of-3 tie-break | Steel-man LLM panels (`decision_source: "llm-panels"`). Unmeasured. `fusion.json` for deterministic decisions is unchanged. | +| Deterministic stages hosted under HarnessX | `stage_processors.py:129-156` (`host_under_harnessx`); only caller is the skipped test `test_slot_contract.py:119-133` | **Retired, no replacement.** ax runs whole tasks, so a model-disabled in-loop host has nothing to host. `stage_processors`/`slot_contract` stay as the standalone deterministic pipeline that `test_gate.py:147-148` pins | Nothing. The function has no production caller. | +| LLM evolution proposer (`MetaAgent.evolve`) | docstring only: `improve/evolve.py:8-9`, `harness_ext.py:7-8` | **Carried by the memory layer** (Requirement 6): `improve/memory.py` rules over the store, fed into `run_round` as extra proposals | Nothing implemented is lost. | + +No CLI flag is HarnessX-specific, so Requirement 1.3 applies to config keys. A key that asked for a retired +capability makes `load_config` raise `RetiredConfigError(ValueError)`, and the CLI prints the message and exits 2. +A silent downgrade is not allowed: + +| Key | Message (abridged) | +| --- | --- | +| `[models] verifier` | "`[models] verifier` was retired with HarnessX on : LLM verification runs on the plane (`verify` + `agree`, `ousast plane run`, see ops/ax/README.md). Remove the key." | +| `[fusion] panel_model`, `[fusion] decider_model` | "... LLM fusion panels were retired; fusion is deterministic; independent LLM agreement is the plane's `agree` task ..." | + +`[harnessx]` is different (Requirement 3.3). It only tuned budgets and never requested a capability, so it loads +with **one** `logging.warning` on logger `openultrasast.config` that names the plane ("`[harnessx]` is ignored: +HarnessX was removed; agentic work runs on the ax plane with per-task budgets, see ops/ax/README.md"). The section +is otherwise ignored. `[models] hunter` keeps its meaning (tool hunter, obligations intent adjudication at +`cli.py:1237`), so it gets no message. With the extra installed it no longer starts a second, HarnessX hunter. The +release notes say so. + +Requirement 1.2 (moved capabilities have a CLI path, a Model per task and a fixture test) is met by the existing +`verify`/`agree` tasks (`ousast plane run`; Model `deepseek-flash` bound per Task; `tests/test_plane_verify.py`, +`tests/test_plane_scoring_parity.py`, `tests/test_plane_tiebreak.py`), and by the new memory and loop tasks below, each with its own fixture test. No +capability is moved into a new Model-bound task. + +### 2. Removal mechanics (Requirement 2) + +**Source.** + +- Delete `harness_ext.py` (97 lines), `hunter_harness.py` (197) and `verify_judge.py` (130). +- `cli.py`: drop the imports at :44, :46 and :94. Drop `harnessx_present`, `hx_hunter` and `hx_verify` (:400-406), + the orchestrator branch and its degradation (:438-456), and the verify degradation (:499-502). Call + `verify_findings(findings)` at :503-511. Drop `hx_fusion` and its degradation (:521-527). Call + `fuse_findings(findings, verifications, high_assurance=config.fusion.high_assurance)`. The `verifier_model` + local goes away. +- `fusion.py`: drop the import at :24, the `panel_model`/`decider_model`/`provider`/`use_harnessx` parameters, and + `_with_degradation`, `_fuse_with_llm_panels`, `_run_llm_panels` and `_text` (:229-338). Rewrite the module + docstring (:10-14) and update `__all__`. +- `stage_processors.py`: drop `host_under_harnessx` (:129-156) and its `__all__` entry, and rewrite the module + docstring (:10-11). `slot_contract.py:7-14` becomes "the zero-dependency substrate for the deterministic + pipeline". +- `config.py`: drop `HarnessxConfig` (:99-107), `ResolvedConfig.harnessx` (:201), `_load_harnessx` (:345-351) and + its call (:230). Drop `ModelConfig.verifier` (:37), `FusionConfig.panel_model/decider_model` (:123-124) and + their loaders (:266, :366-367). Add `RETIRED_KEYS` and the one-warning rule above. `harness.py:183` keeps the + `"verifier": None` literal in `model_roles`, so `harness.json` keeps its schema byte for byte (the key has always + been `null` in the baseline config). +- Comment-only mentions: `gate.py:174` ("a model-backed stage"), and `improve/evolve.py:8-9`, which now names the + memory proposer. +- `plane/Dockerfile.runner:23` loses "no HarnessX". + +**Packaging.** + +- Delete the `harnessx` extra and its comment (`pyproject.toml:14-22`) and the `harnessx.*` mypy override and its + comment (:67-71). +- `uv lock` regenerates `uv.lock`. The `harnessx` package (:1062), the extra (:2145-2146), the requirement + (:2169) and `provides-extras` (:2178) go. +- Extras that remain: `semantic`. There is no `all` extra. `pip install .` and `pip install '.[semantic]'` in a + fresh venv are the Requirement 2.2 check. +- No source or test imports a HarnessX transitive dependency. The only third-party imports are `yaml`, + `tree_sitter` and `pytest` (checked with an import scan), so dropping the lock entry cannot break an import. + +**Tests** (14 files plus `conftest.py`): + +| File | Action | +| --- | --- | +| `test_harness_ext.py`, `test_hunter_harness.py`, `test_verify_judge.py`, `test_cli_hx_dispatch.py` | **Delete.** Each exercises only HarnessX or its dispatcher. | +| `conftest.py` | Rename fixture `assert_cold_of_harnessx` to `assert_cold_import(code)`: a fresh interpreter runs `code`, with no module-name guard. `test_semantic_extra.py:28-35` is its remaining user (the tree-sitter check brings its own assertion). | +| `test_gate.py` | Delete `test_harnessx_stage_stays_within_tolerance_of_baseline` (:124-150) and the `has_harnessx` import (:23). Replace the two "never import" tests (:155-175) with `test_governance_planes_import_only_stdlib_and_declared_dependencies`, an allow-list (stdlib, `openultrasast`, `yaml`) over module-level imports of `GOVERNANCE_SCORING_BENCHMARK_PLANES`. It covers any extra without naming one. Update the docstring at :5-6. | +| `test_scan_pipeline_integrity.py` | Rename `test_standard_scan_extra_absent_matches_deterministic_baseline` to `test_standard_scan_with_a_hunter_model_keeps_deterministic_findings` (config `hunter` only). Rename `test_quick_scan_is_unaffected_by_the_extra` to `test_quick_scan_records_no_degradation`. Drop the `has_harnessx` monkeypatch and the `harnessx` parameter. Recompute the expected degradation set from the post-removal run, since `hunter_pool`/`verify` entries no longer exist. | +| `test_slot_contract.py` | Delete `test_host_under_harnessx_disables_the_model` and the imports at :16 and :22. Keep the rest. | +| `test_config.py` | Replace `test_harnessx_defaults_when_section_absent` and `test_load_config_reads_harnessx_section` with `test_retired_agentic_section_loads_with_one_warning` (caplog: exactly one record naming the plane) and `test_retired_llm_keys_fail_naming_the_replacement` (verifier, panel_model, decider_model). | +| `test_fusion.py` | Delete `test_dispatch_records_degradation_when_llm_requested_but_extra_absent` (:142-146) and the import at :16. Rename the dispatch calls to `fuse_findings`. | +| `test_tool_hunter.py` (:229, :266), `test_scan_deep.py` (:144) | Drop the `has_harnessx` monkeypatch line. | +| `test_semantic_extra.py` | Use `assert_cold_import`. | +| `test_module_audit.py` | The subsystem tuple at :31 becomes `("fusion", "mcp", "skills")`. A new assertion checks that the three deleted modules are absent from the audit. | + +**Module audit.** `model/audit.py:76-77` (`harness_ext`, `hunter_harness`) are deleted, and `:78` +(`stage_processors`) is reworded to "deterministic slot-contracted scan stages; pinned by the zero-dependency +guard in test_gate". Regenerate `benchmarks/measurements/2026-09-08-module-audit.json` in place with `to_dict(audit(SRC))`. +`test_the_manifest_matches_the_tree` pins that path. The diff shows three rows removed and no orphan. + +**Search (Requirement 2.4).** `grep -rni harnessx src tests pyproject.toml README.md` may match only lines tagged +`retired 2026-..`: + +- the `RETIRED_KEYS` entry and warning text in `config.py` +- the two retirement tests in `test_config.py` +- the one README line in the plane section + +Requirement 3.3 cannot be implemented or tested without naming the section, so these lines count as retirement +notes. The check is a test (`test_no_harnessx_outside_retirement_notes`) that fails on any untagged match. It +builds its pattern at runtime, so it does not match itself. + +### 3. Equality proof (Requirement 3) + +The spec's "`ousast regress`" is not a subcommand (`cli.py:133-251`). Regress exists only as the deep-mode +sandbox stage. On this project the regression command over the development pins is `ousast pairs` +(README.md:38): the vuln-vs-fixed replay over local fixtures and vendored pinned VFC pairs, offline by default. +The benchmark replay is README.md:35's manifest. The baseline also covers the two paths the removal touches +directly: one standard-mode scan (verify and fusion) and one `improve --dry-run` (evolve). Everything is +quick or standard mode with no Joern on the host, so each command takes minutes, not hours, on this 7 GB host. + +**Environment (instrument first).** + +- Run from a `git archive HEAD` export in the scratchpad (the source is frozen, per the long-runs rule). +- `uv sync --frozen --extra semantic`, which drops the installed `harnessx`. It is installed in `.venv` today. +- `OPENULTRASAST_SKIP_DOTENV=1` and `env -u DEEPSEEK_API_KEY -u OPENROUTER_API_KEY -u ANTHROPIC_API_KEY -u OPENAI_API_KEY`, + plus an empty `--config`, so no command can reach a model. + +The harness asserts `find_spec("harnessx") is None`, `find_spec("tree_sitter") is not None`, and a +`uv pip freeze` hash. It also prints the byte size of each input manifest, and fails on any non-zero exit code. + +**Commands** (one script, `benchmarks/harnessx_removal_equality.py record|compare `): + +1. `pytest -q -p no:cacheprovider --junitxml=suite.xml`, which gives outcome per test id. +2. `ousast pairs --json > pairs.json`. Instrument check: pair count > 0 and at least one TP. +3. `ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick`. Keep + `benchmark_result.json`, `calibration_records.json` and `external_baseline_deltas.json`. Instrument check: + `expected > 0`. +4. `ousast scan benchmarks/fixtures/python-vulnerable --mode standard --config empty.toml`. Keep + `findings.json`, `verification.json`, `fusion.json`, `report.sarif`, `harness.json` and `manifest.json`. +5. `ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --no-pair-gate`, stdout plus + the dry-run journal. + +**Stored.** `benchmarks/measurements/-harnessx-removal-baseline/`: the normalized outputs, and `env.json` +(commit, freeze hash, instrument checks). The script's exclusion list is written into that directory's +`README` line. + +**Comparison.** JSON is parsed, normalized and re-serialized with sorted keys. Everything else is compared as +raw bytes. Excluded: + +- keys `runtime_seconds`, `scan_id`, `timestamp`, `started`, `finished`, `created_at`, `generated_at`, and any + key ending `_seconds` or `_ms` +- run directory names (timestamped, `benchmark.py:186`) +- `trace/events.jsonl` as a whole (timings) +- the export's absolute root path, replaced by `` + +For the suite, every test id that survives must keep its outcome. Deleted and renamed ids are compared through +the map in the Tests table above, and any other difference is a failure. Any difference in 2-5 blocks the merge. + +### 4. Memory layer (Requirement 6.1, 6.4) + +**Store** (`plane/memory.py`). The store lives at `results_root()/memory/` (`~/ousast-results/plane/memory/`, +honouring `OUSAST_RESULTS`, `reconciler.py:58-59`). It is JSON files, not SQLite: the rest of the plane is JSON, +diffable and greppable, and one run at a time holds the run lock, so SQLite's concurrency buys nothing. + +``` +memory/ + index.jsonl one row per ingested (run, task): sha256 of its memory.jsonl, row count + facts/.json facts.json by content (identical facts stored once) + repos/____/.jsonl the rows of one repository + 40-hex pin +``` + +Ingest is idempotent: an `index.jsonl` hit skips the delivery, and rows carry a deterministic `id`. Ingest also +refuses to write when `shutil.disk_usage` reports less than 1 GiB free. The disk is at 98% today, so it names +the store path in the error. + +**`remember` task** (`plane/tasks/remember.py`, no Model, budget `{usd: 0, calls: 0}`, one per case). It reads the +case's delivered artifacts as inputs: `facts.json`, the `units.jsonl` of `va`/`vb`/`vc`, `final/agreed.json` and +`final/disputed.json` with the per-candidate rows, and `alerts.jsonl`. It takes the repository URL and pin from +the bound Workspace (`AX_WORKSPACES_YAML`, `OUSAST_GIT_PINS`) and the population from the Run annotation +`openultrasast.io/population`. It writes `memory.jsonl` and passes `facts.json` through. Task code interprets +the artifacts, so the reconciler and runner stay content-blind (ai-service-plane Req 3.4). After `reconciler.run` +returns, `cli._plane` calls `memory.ingest(run_dir)`. `ousast plane remember ` does the same by hand. + +**Row kinds** (every row has `id`, `kind`, `repo`, `pin`, `run`, `task`, `population`, `split`, `image`): + +- `facts`: `sha256`, `candidates_digest` (`repo_facts._digest`, `repo_facts.py:188`), `files`. +- `verdict`: `candidate` (`path::function`), `family`, `site`, `passes {a, b, c}` (true/false/null), `final` + (`agreed | rejected | disputed`), `tiebreak` (bool), `site_match` (bool/null), `usd`, `turns`. +- `unit_cost`: `pass`, `path`, `usd`, `calls`, `usage`, `turns`, `model`. +- `alert`: `rule_id`, `rule_status` (`enabled | shadow`), `path`, `line`, `function`, `pin_role` + (`vulnerable | fixed`). +- `proposal_outcome`: `edit_key`, `evidence_digest`, `outcome` (`accepted | reverted | rejected`), `gate_run`. + +**Fact reuse.** A facts entry is valid for the key (repo, pin, candidates digest, runner image digest). The image +is in the key because the host's source and the image's source can differ, and facts computed by other code are +not the same facts. `ousast plane workspaces --validation-set` writes the key as the Task annotation +`openultrasast.io/memory-key`. Annotations are already stripped before ax (`manifests.py:4-6`). Before +`reconciler.run`, `cli._plane` calls `memory.seed(run_manifest)`. For each Run entry whose Task carries a key with +stored facts and no state yet, seed writes `//facts.json` and a `summary.json` +(`status: done, units_done == units_total == 1, reused: {"sha256", "from_run"}`). It then calls +`reconciler.mark_done(run, task, reused=...)`, a public helper of about 10 lines that takes the run lock, sets +the state and releases it. The reconciler already skips `done` tasks (`reconciler.py:342`). The receiver already +serves any regular file under the run directory as a declared input (`egress.py:242-246`). A changed pin, a +changed candidate set or a rebuilt image recomputes. + +### 5. Proposals from memory (Requirement 6.2, 6.4) + +**Interface.** The proposal surface HarnessX would have used is `run_round` (`improve/evolve.py:149-247`). It +builds `RuleStatusEdit`s (`improve/validator.py:36-47`), filters them through the journal's novelty gate +(`journal.reverted_edit_keys`, `journal.py:26-38`), validates each with `EvolveValidator.validate` +(`validator.py:75-81`, which accepts only `RuleStatusEdit` and `PolicyConstantEdit`), smoke-replays and gates. + +`improve/memory.py` exposes: + +``` +propose_from_memory(rows, ruleset_by_id, current_ledger, journal_rounds, *, excluded) -> list[MemoryProposal] +MemoryProposal = (edit: RuleStatusEdit, provenance: dict) +``` + +`run_round` gains `proposals: Sequence[MemoryProposal] = ()`. It appends their edits after the benchmark's edits +and keeps the first edit per `key()`. The validator and the gate are untouched. `ousast improve` gains +`--memory [DIR]` (off by default, so the deterministic `improve` output is byte-identical) and +`--qualify-population NAME` (repeatable). + +**Rules** (the thresholds are module constants and part of the provenance record). Join key: an `alert` and a +`verdict` match when repo, pin and `path::function` are equal. A `disputed` final counts as no evidence in +either direction. + +- **M1, repeated false alerts** (`enabled -> shadow`). Rule R qualifies when both hold: + - at least 3 R alerts sit at candidates whose final is `rejected`, or on a `fixed` pin inside the case's declared + fix range, across at least 2 distinct repositories; + - no R alert sits at an `agreed` candidate or a `site_match` site. +- **M2, missed declared sites** (`shadow -> enabled`). Shadow rule R qualifies when all three hold: + - its shadow alerts hit at least 2 `agreed` candidates with `site_match == true`, across at least 2 + repositories; + - no enabled rule alerts at those candidates; + - R has no alert on a `fixed` pin at the same function and at most 1 alert at a `rejected` candidate. +- **M3, frequently disputed families.** Families with at least 5 candidates and a dispute rate of at least 0.3 go + to `signals.json` as advisory input for plane routing (for example, scheduling pass `c` up front). They are not + an edit and never reach the validator. + +**Provenance.** Each proposal's `rationale` is `memory:`, where `proposal_id` = sha256 of +(rule, edit key, sorted evidence row ids, thresholds). The sidecar `memory_proposals.jsonl`, next to the journal, +holds: + +- `proposal_id`, `edit_key`, `rule_id` +- `evidence` (the row ids plus their repo, pin, run and task) +- the counts that met each threshold +- `excluded` (populations and pins removed by the guard, with counts) +- the store's `index.jsonl` digest at proposal time + +`evolve._record` (`evolve.py:366-396`) adds `"evidence": proposal_id` to an edit's journal entry only when the +edit came from memory. Otherwise the journal is unchanged byte for byte. A reader follows journal, then sidecar, +then store rows, then the run directory. + +**Novelty.** `journal.reverted_edit_keys` does not block a rule edit that carries a rationale (`journal.py:36`), so +a memory proposal would otherwise come back every round. The proposer itself skips any key whose reverted journal +entry has the same evidence digest. New evidence is the "new rationale" that allows a retry. + +**Train-on-test guard.** A row is excluded from proposals when any of these holds: + +- (a) its `population` is one of `--qualify-population`, which is also where the proposal will be gated or + qualified; +- (b) its (repo, pin) is a case of the benchmark manifest being gated; +- (c) its (repo, pin) is a `split = "holdout"` pair of the pair catalog, which the per-profile holdout clause + (`evolve.py:204-214`) measures. + +This is the rule the closed-loop leak broke, where a holdout pair taught the shape that "recovered" it. Here the +filter runs before any rule sees a row. Concretely, rows from `validation-46` (population +`independent-v2/validation`) never propose edits that are then qualified on `independent-v2/validation`. The +tie-break was designed on that set, so it is development evidence only. + +### 6. The loop as a plane Run (Requirement 6.3) + +`ousast plane workspaces --validation-set ... --loop` extends the generated Run. It also emits a Workspace for each +case's fixed pin (the population's `fixed` sha, already read by `generate.fix_ranges`, `generate.py:89`): + +| Task (per case unless noted) | Model | Budget | Inputs -> outputs | +| --- | --- | --- | --- | +| `-facts`, `-va`, `-vb`, `-agree`, `-vc`, `-final` | verify passes: `deepseek-flash`; others none | as generated today (`generate.verify_budget`) | unchanged | +| `-alerts` (new) | none | `{0, 0}`, `serialize: engine` | vulnerable + fixed Workspaces -> `alerts.jsonl` (quick-mode `quick_scan_findings` with enabled and shadow rules, function-attributed) | +| `-remember` (new) | none | `{0, 0}` | facts, va, vb, vc, final, alerts -> `memory.jsonl`, `facts.json` | +| `loop-measure` (once) | none | `{0, 0}` | every `remember` + `memory-snapshot/rows.jsonl` -> `measure.json` (the increment metrics: declared sites agreed, cost per candidate, dispute rates) | +| `loop-propose` (once) | none | `{0, 0}` | measure, snapshot, journal -> `proposals.jsonl`, `memory_proposals.jsonl`, `signals.json` | +| `loop-improve` (once) | none | `{0, 0}`, `serialize: engine` | proposals; Workspace = this repository at the commit under test -> `gate.json` (the `RoundOutcome`), `journal.json`, `rule_policy.json` (only when accepted) | + +Before the Run starts, `memory-snapshot` is seeded into the run directory like reused facts. It holds the store +rows after the guard, since a sandboxed task cannot read the host store. A `{usd: 0, calls: 0}` budget makes +`budget.py` refuse any call, so a model-free task that tries one fails loudly rather than spending. Attribution +per task is the existing `ousast plane status` table. + +**The gate's verdict** lands in `~/ousast-results/plane//loop-improve/gate.json` and is ingested as +`proposal_outcome` rows. An accepted `rule_policy.json` is adopted into the repository only by a maintainer +commit, as any ledger change is today (README "Rule-level loop"). + +### 7. Documentation and specs (Requirement 4) + +- **README.** Replace "Configuring the HarnessX agentic plane" (README.md:293-329) with "The agentic plane (ax)": + what runs there (repo-facts, verify, agree, memory), `ousast plane run|status|remember`, a pointer to + `ops/ax/README.md`, and one tagged line saying HarnessX was retired. Rename "The HarnessX self-improving cycle" + (:330) to "The self-improving cycle". Its calibration ledger and `ousast improve` content is not HarnessX and + stays. The MetaAgent paragraph (:393-400) is replaced by the memory proposer (`--memory`, provenance, guard). + Drop the `panel_model`/`decider_model` lines at :227-228. +- **`docs/examples.md` section 6** becomes "Running the agentic plane", with the plane doctor, run and status + commands. +- **`docs/threat-model.md:17-20, 32-34`**: egress is the per-task EgressPolicy (receiver, Git hosts, Model + `egress-hosts`, passthrough only). Credentials travel only in the start request. Spend is the per-task + `usd`/`calls` budget in the Run. +- **Retirement notes.** A dated note ("Retired 2026-..: HarnessX was removed by `harnessx-removal`; the agentic + plane is ax (`ai-service-plane`), proposals come from the plane memory. History below is unchanged.") goes at + the top of the requirements (and design, where present) of `harnessx-self-improving-rulesets`, + `learning-harness`, `three-stage-scan` and `openrouter-sast-harness`, and in `.kiro/steering/overview.md` near + :175. The other specs that mention HarnessX in passing keep their history untouched. +- `RELEASE_NOTES.md` gets one entry listing the retired keys. Kiro tooling stays out of user docs. + +## Data Models + +- `memory.jsonl` (remember output) and `repos//.jsonl`: the rows above, one JSON object per line, + sorted keys, stable `id` = sha256(kind, run, task, candidate|path|rule_id+line). +- `index.jsonl`: `{"run", "task", "sha256", "rows", "ingested"}`. +- `memory_proposals.jsonl`: `{"proposal_id", "edit_key", "rule_id", "rule": "M1|M2", "evidence": [...], + "counts": {...}, "thresholds": {...}, "excluded": {...}, "store_digest"}`. +- Journal edit entry: the existing fields (`evolve.py:381-384`) plus `"evidence"` only for memory edits. +- Equality record: `env.json` `{"commit", "freeze_sha256", "harnessx_importable": false, "tree_sitter": true, + "inputs": {path: bytes}, "exclusions": [...]}`. + +## Error Handling + +- `RetiredConfigError`: exit 2 with the key and its replacement, before any stage runs. +- A `[harnessx]` section produces exactly one warning, and loading continues. +- `memory.ingest` on a run without `remember` outputs is a no-op that says so. A malformed row fails the ingest + and names the file and line, and nothing partial is written: rows are staged in a temporary file and renamed. +- A low-disk refusal names the store and the free space. The plane run's result is unaffected, and ingest can be + repeated with `ousast plane remember`. +- `seed` never overwrites an existing task directory or a non-pending state. A facts file whose sha256 no longer + matches is dropped from the store with a warning, and that task recomputes. +- `propose_from_memory` with an empty store after the guard returns no proposals, and `run_round` reports + `no_proposals` as today. +- The equality script exits non-zero on the first failed instrument check. It also exits non-zero on any + difference outside the exclusion list and prints the first differing path and key. + +## Testing Strategy + +- `tests/test_config.py`: the one-warning section and the retired keys (above). +- `tests/test_no_harnessx_references.py`: the tagged-only search of Requirement 2.4. +- `tests/test_plane_memory.py`: ingest is idempotent and content-addresses identical facts. `seed` marks a facts + task done and serves its file, and does not for a changed pin, candidate digest or image. Low-disk refusal + (monkeypatched `disk_usage`). +- `tests/test_plane_remember.py`, `test_plane_alerts.py`, `test_plane_loop.py`: fixture run directories built from + the recorded validation-46 shapes. `remember` rows match `agree`'s per-candidate rows. `alerts` attributes + functions. `measure` reproduces the increment's recorded metrics on the recorded artifacts. `propose` and + `improve` work end to end on a fixture manifest. +- `tests/test_improve_memory.py`: + - M1 and M2 fire at their thresholds and not below. + - A disputed candidate counts neither way. + - The guard removes rows by population, manifest case and holdout pair, so a proposal that exists only because + of a holdout row disappears. + - Provenance ids are stable across runs. + - A reverted memory edit is not re-proposed on the same evidence, and is re-proposed on new evidence. + - `run_round` without proposals writes a journal byte-identical to today's. +- `tests/test_plane_reconciler.py`: `mark_done` under the lock. The existing under-500-lines test stays. +- The existing gate and pair tests, `test_benchmark_output_is_byte_identical_to_golden_baseline` + (`test_gate.py:104`) and the equality script (section 3) are the regression net. + +## Requirement Traceability + +| Requirement | Design section | Verified by | +| --- | --- | --- | +| 1.1 fate of every capability | 1 (table) | this document; commit 5 | +| 1.2 moved capabilities reachable, Model per task, fixture test | 1 (existing `verify`/`agree`), 4-6 (new tasks are Model-free) | `test_plane_verify.py`, `test_plane_tiebreak.py`, new plane task tests | +| 1.3 retired keys fail naming the replacement | 1 (`RetiredConfigError`) | `test_config.py` | +| 2.1 modules and guards gone | 2 Source | commit 6; audit test | +| 2.2 extra, override, lock gone; installs succeed | 2 Packaging | fresh-venv `pip install .` / `'.[semantic]'` | +| 2.3 tests deleted or renamed; audit regenerated | 2 Tests, Module audit | `test_module_audit.py` | +| 2.4 search returns only retirement notes | 2 Search | `test_no_harnessx_references.py` | +| 3.1 baseline without the extra | 3 | commit 1 record | +| 3.2 byte-identical after, exclusions written | 3 Comparison | commit 7 record | +| 3.3 `[harnessx]` loads with one warning | 1 | `test_config.py` | +| 4.1-4.3 docs, retirement notes, Kiro out of user docs | 7 | commit 8 review | +| 5.1-5.2 order, green steps, no deletion before replacement | Commit sequence | per-commit gates | +| 6.1 store keyed by repo + pin, fact reuse | 4 | `test_plane_memory.py`, `test_plane_remember.py` | +| 6.2 deterministic proposals through validator and gate | 5 | `test_improve_memory.py` | +| 6.3 loop as a plane Run with per-task Models and budgets | 6 | `test_plane_loop.py`; `ousast plane status` attribution | +| 6.4 provenance and train-on-test guard | 5 | `test_improve_memory.py` | + +## Commit sequence (Requirement 5), mapped to future tasks + +Each commit is gated on the full suite's own exit code, `ruff` and `mypy`. + +1. **Baseline** (3.1): the equality script and `benchmarks/measurements/-harnessx-removal-baseline/`, + recorded with the extra uninstalled. +2. **Memory store and `remember`** (6.1): `plane/memory.py`, `tasks/remember.py`, `reconciler.mark_done`, seed and + ingest wiring in `cli._plane`, generator annotations, fixture tests. +3. **Memory proposer** (6.2, 6.4): `improve/memory.py`, `run_round(proposals=...)`, `ousast improve --memory`, + sidecar and journal evidence, guard tests. +4. **Loop Run** (6.3): `alerts`, `measure`, `propose` and `improve` tasks, `--loop` generation, fixed-pin + Workspaces, fixture tests. +5. **Retire the scan paths** (1, 3.3): the `cli.py`, `fusion.py`, `verify_judge.py` and `stage_processors.py` + changes, `RetiredConfigError`, the one warning, and the test renames that follow. At this point nothing imports + `harness_ext`/`hunter_harness`. Their replacements (plane tasks, and the memory proposer from 2-4) have landed. +6. **Delete code and packaging** (2): the two modules, the extra, the mypy override, `uv.lock`, the deleted tests, + the fixture rename, audit rows and the manifest regeneration, and the reference search test. +7. **Equality proof** (3.2): rerun the script, commit the comparison record next to the baseline, and link it in + the commit message. +8. **Docs and specs** (4): the README, `docs/examples.md`, `docs/threat-model.md`, retirement notes, steering and + release notes. + +## Risks + +- **Existing users of the LLM scan paths.** There is no `--llm` flag. Users who set `[models] verifier` or + `[fusion] panel_model` get a hard error, which is intended (1.3). Users who installed the extra and set + `[models] hunter` lose the second (HarnessX) hunter without an error. Only the release note covers that, + because the key is still valid. No measurement suggests either path found anything the remaining paths miss. + The absence of a measurement is itself the risk. +- **Plane precision is unsolved.** `plane-increment-2.json` agrees 28 candidates, of which 16 are declared + sites. The other 12 are unadjudicated and several may be false. M1 treats a `rejected` final as negative + evidence, while plane recall is 16/20, so "rejected" is not "safe". The thresholds (2 or more repositories, zero + agreed hits) and the unchanged recall/FP gate are the guard. Precision work (judge design) remains out of scope. +- **Evidence is thin.** Today's store would hold 15 cases from one validation population, which is exactly the + population the guard excludes for its own qualification. Expect `no_proposals` until a second population has + run through the loop. +- **Disk.** The home volume is 98% full (2.4 GB free). One validation-46-sized Run adds about 3 MB of facts + (content-addressed; about 0.1 MB when pins are unchanged) plus about 0.5 MB of rows. Ingest refuses below 1 GiB + free. The store has no eviction in this spec. +- **The baseline measures the environment as well as the code.** `.venv` has `harnessx` installed today. A + baseline taken without re-syncing would include it. The script's `find_spec` and freeze checks exist for that. +- **Seeding writes reconciler state from outside the reconciler.** It is confined to `mark_done` under the run + lock. The reconciler's line budget (465 of 500) leaves no room for fact reuse inside the reconciler. diff --git a/.kiro/specs/harnessx-removal/spec.json b/.kiro/specs/harnessx-removal/spec.json index 9f64655..c60b559 100644 --- a/.kiro/specs/harnessx-removal/spec.json +++ b/.kiro/specs/harnessx-removal/spec.json @@ -1,16 +1,16 @@ { "feature_name": "harnessx-removal", "created_at": "2026-09-30T08:00:00Z", - "updated_at": "2026-09-30T09:30:00Z", + "updated_at": "2026-09-30T10:30:00Z", "language": "en", - "phase": "requirements-approved", + "phase": "design-generated", "approvals": { "requirements": { "generated": true, "approved": true }, "design": { - "generated": false, + "generated": true, "approved": false }, "tasks": {