Open_Ultra_SAST/.kiro/specs/harnessx-removal/requirements.md

Requirements Document

Introduction

HarnessX is the optional agentic plane (openultrasast[harnessx]): an LLM hunter pool, an LLM judge and LLM fusion panels, each behind a capability guard with a deterministic fallback, plus a lazy bridge that hosts deterministic stages under a HarnessX processor. The ai-service-plane increment met its gate on google/ax (benchmarks/independent/plane-increment-2.json), which is the condition the maintainer set on 2026-09-29: "goal is to remove harnessx and to shift to ax fully once we establish we have value." This feature removes HarnessX from the product and moves every LLM capability it carried either onto the plane or into a recorded retirement. Inventory: brief.md (verified 2026-09-30).

Maintainer, 2026-09-30: "Please remove harnessx completely. the memory layer that saves tokens takes over the evolutionary improvements as well as loop engineering." So nothing of HarnessX stays, not even as an optional path, and the self-improving cycle HarnessX was meant to drive (the LLM MetaAgent.evolve proposer plugged into improve/evolve.py) is carried by the plane's memory layer instead: what runs record (repository facts, verdicts, disputes, costs) becomes the input to proposals and to the run-measure-propose-gate loop.

Boundary Context

• In scope: the HarnessX-only modules (harness_ext.py, hunter_harness.py); the guarded branches in cli.py (scan --llm path, HxScanOrchestrator), verify_judge.py (judge path), fusion.py (LLM panels), stage_processors.py (host_under_harnessx), config.py (HarnessxConfig); the harnessx extra, its mypy override and lock entry; the tests that exercise HarnessX; README and docs sections; dated retirement notes in the specs and steering that name HarnessX.
• Also in scope: the plane memory layer as the source of evolutionary proposals and the loop (Requirement 6), replacing the HarnessX proposer.
• Out of scope: changing the deterministic engine, rulesets or evolve's validator and gate (proposals change source, not how they are validated and gated); new detection logic; qualification on an independent population; the plane's own precision work (judge design) -- where a capability moves to the plane, this spec wires it, it does not improve it.
• Adjacent expectations: every deterministic path (quick mode, regress, benchmark, gate, evolve) produces byte-identical output before and after; vulnerabilities-over-plumbing applies, so capabilities are kept or retired by what they detect, reported in TP/FP terms where a measurement exists.

Requirements

Requirement 1: HarnessX is removed completely; the plane is the only agentic path

User Story: As the maintainer, I want no HarnessX path left, optional or not, and every LLM capability either on the plane or explicitly retired, so that nothing is lost silently.

Acceptance Criteria

1. No HarnessX code path remains, optional or guarded. The design lists each capability -- LLM hunter pool (ousast scan --llm), LLM judge (verify_judge), LLM fusion panels (fusion), deterministic stages hosted under HarnessX (host_under_harnessx), the LLM evolution proposer -- with its fate: carried by a plane task (named), carried by the memory layer (Requirement 6), or retired with the deterministic path kept.
2. A capability moved to the plane is reachable from the CLI through the plane (ousast plane run with a documented Run, or a CLI flag that submits one), binds its own Model per task (ai-service-plane Req 7), and has a fixture test like the other plane tasks.
3. A retired capability's CLI flags and config keys fail with a message that names the replacement or says it was retired, never a silent no-op.

Requirement 2: HarnessX code, packaging and tests are gone

User Story: As the maintainer, I want no HarnessX code path left to maintain.

Acceptance Criteria

1. harness_ext.py and hunter_harness.py are deleted; no module imports harnessx; use_harnessx parameters, has_harnessx checks and host_under_harnessx are removed; the deterministic fallbacks become the only code path where a capability is retired.
2. The harnessx extra, the mypy override for harnessx.* and the lock entry are removed; pip install . and pip install '.[all]' (or whatever extras remain) succeed.
3. Tests that only exercise HarnessX are deleted; tests that assert fallback behaviour are kept and renamed to describe the behaviour, not the absence of HarnessX; the module audit is regenerated with no orphan.
4. A search for harnessx (case-insensitive) in src/, tests/, pyproject.toml and the README returns only the retirement notes of Requirement 4.

Requirement 3: Deterministic behaviour is unchanged, and proven so

User Story: As the maintainer, I want evidence, not assurance, that removing HarnessX changed nothing a deterministic run produces.

Acceptance Criteria

1. Before any deletion, a baseline is recorded without the extra installed: the full host suite, and the outputs of ousast regress and one ousast benchmark replay on the development pins, stored under benchmarks/measurements/.
2. After the removal the same commands produce byte-identical outputs (timestamps and durations excluded, and the exclusion list is written down); any difference blocks the merge.
3. A config file that still carries a [harnessx] section loads with one warning that names the plane, and the section is otherwise ignored.

Requirement 4: Documentation and specs tell the new story

User Story: As a reader of the README or a spec, I want to learn that the agentic plane is ax, not HarnessX.

Acceptance Criteria

1. The README sections "Configuring the HarnessX agentic plane" and "The HarnessX self-improving cycle" are replaced by a short section on the ax plane that points at ops/ax/README.md; docs/examples.md section 6 and docs/threat-model.md (egress and spend caps) describe the plane's egress policy and budgets instead.
2. Specs that name HarnessX (harnessx-self-improving-rulesets, learning-harness, three-stage-scan, openrouter-sast-harness) and .kiro/steering/overview.md get a dated retirement note at the top; their history is not rewritten.
3. Maintainer-only Kiro tooling stays out of user docs (standing rule).

Requirement 5: The removal lands in one reviewable sequence

User Story: As the maintainer, I want to review the removal in steps that each leave main green.

Acceptance Criteria

1. Order: baseline (3.1) -> capabilities moved or retired (1) -> code and packaging removed (2) -> equality proof (3.2) -> docs and specs (4). Each step is a commit gated on the full suite's own exit code, ruff and mypy.
2. No step deletes code whose replacement (Requirement 1) has not landed and passed its fixture test.

Requirement 6: The memory layer carries evolutionary improvement and loop engineering

User Story: As the maintainer, I want the token-saving memory layer to be what the system learns from, so that improvement proposals and the improvement loop come from recorded evidence instead of a HarnessX agent.

Acceptance Criteria

1. Plane runs persist their memory beyond the run directory in one store keyed by repository and pin: repository facts (repo-facts), per-candidate verdicts per pass with the final decision, disputes and tie-breaks, and cost and tool turns per unit; later runs reuse facts for an unchanged pin instead of recomputing them.
2. evolve's proposals are generated from that store (for example: families or rules whose candidates are often disputed, missed declared sites, repeated false alerts), deterministically and without a HarnessX agent; they pass through the existing validator and gate unchanged.
3. The improvement loop -- run, measure, propose, validate, gate -- is expressible as a plane Run whose tasks bind their own Models and budgets (ai-service-plane Req 7), so each iteration's token spend is attributed per task.
4. A proposal records which memory entries it came from, so a later reader can trace every accepted change to the evidence behind it; proposals never train on the population used to qualify them (the train-on-test rule).