mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
spec(harnessx-removal): requirements for review
The ai-service-plane increment met its gate on ax (plane-increment-2.json), the condition for removing HarnessX. Requirements: an explicit fate per capability (moved to a plane task or retired), code/packaging/tests removed, deterministic outputs proven byte-identical against a recorded baseline, docs and specs retold, one reviewable sequence. Brief inventory re-verified. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
df1dfa3b7a
commit
9391a69f25
@@ -0,0 +1,98 @@
|
|||||||
|
# Requirements Document
|
||||||
|
|
||||||
|
## Introduction
|
||||||
|
|
||||||
|
HarnessX is the optional agentic plane (`openultrasast[harnessx]`): an LLM hunter pool, an LLM judge and LLM
|
||||||
|
fusion panels, each behind a capability guard with a deterministic fallback, plus a lazy bridge that hosts
|
||||||
|
deterministic stages under a HarnessX processor. The ai-service-plane increment met its gate on google/ax
|
||||||
|
(`benchmarks/independent/plane-increment-2.json`), which is the condition the maintainer set on 2026-09-29:
|
||||||
|
"goal is to remove harnessx and to shift to ax fully once we establish we have value." This feature removes
|
||||||
|
HarnessX from the product and moves every LLM capability it carried either onto the plane or into a recorded
|
||||||
|
retirement. Inventory: `brief.md` (verified 2026-09-30).
|
||||||
|
|
||||||
|
## Boundary Context
|
||||||
|
|
||||||
|
- **In scope**: the HarnessX-only modules (`harness_ext.py`, `hunter_harness.py`); the guarded branches in
|
||||||
|
`cli.py` (scan `--llm` path, `HxScanOrchestrator`), `verify_judge.py` (judge path), `fusion.py` (LLM panels),
|
||||||
|
`stage_processors.py` (`host_under_harnessx`), `config.py` (`HarnessxConfig`); the `harnessx` extra, its mypy
|
||||||
|
override and lock entry; the tests that exercise HarnessX; README and docs sections; dated retirement notes in
|
||||||
|
the specs and steering that name HarnessX.
|
||||||
|
- **Out of scope**: changing the deterministic engine, rulesets or `evolve`'s validated/gated machinery; new
|
||||||
|
detection logic; qualification on an independent population; the plane's own precision work (judge design) --
|
||||||
|
where a capability moves to the plane, this spec wires it, it does not improve it.
|
||||||
|
- **Adjacent expectations**: every deterministic path (quick mode, `regress`, `benchmark`, `gate`, `evolve`)
|
||||||
|
produces byte-identical output before and after; `vulnerabilities-over-plumbing` applies, so capabilities are
|
||||||
|
kept or retired by what they detect, reported in TP/FP terms where a measurement exists.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
### Requirement 1: Each HarnessX capability is moved or retired by an explicit decision
|
||||||
|
|
||||||
|
**User Story:** As the maintainer, I want every LLM capability HarnessX provided to have a recorded fate, so that
|
||||||
|
nothing is lost silently.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. The design lists each capability -- LLM hunter pool (`ousast scan --llm`), LLM judge (`verify_judge`), LLM
|
||||||
|
fusion panels (`fusion`), deterministic stages hosted under HarnessX (`host_under_harnessx`) -- with one of:
|
||||||
|
moved to a plane task (named), or retired (with the evidence or reason).
|
||||||
|
2. A capability moved to the plane is reachable from the CLI through the plane (`ousast plane run` with a
|
||||||
|
documented Run, or a CLI flag that submits one), binds its own Model per task (ai-service-plane Req 7), and
|
||||||
|
has a fixture test like the other plane tasks.
|
||||||
|
3. A retired capability's CLI flags and config keys fail with a message that names the replacement or says it
|
||||||
|
was retired, never a silent no-op.
|
||||||
|
|
||||||
|
### Requirement 2: HarnessX code, packaging and tests are gone
|
||||||
|
|
||||||
|
**User Story:** As the maintainer, I want no HarnessX code path left to maintain.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. `harness_ext.py` and `hunter_harness.py` are deleted; no module imports `harnessx`; `use_harnessx`
|
||||||
|
parameters, `has_harnessx` checks and `host_under_harnessx` are removed; the deterministic fallbacks become the
|
||||||
|
only code path where a capability is retired.
|
||||||
|
2. The `harnessx` extra, the mypy override for `harnessx.*` and the lock entry are removed; `pip install .` and
|
||||||
|
`pip install '.[all]'` (or whatever extras remain) succeed.
|
||||||
|
3. Tests that only exercise HarnessX are deleted; tests that assert fallback behaviour are kept and renamed to
|
||||||
|
describe the behaviour, not the absence of HarnessX; the module audit is regenerated with no orphan.
|
||||||
|
4. A search for `harnessx` (case-insensitive) in `src/`, `tests/`, `pyproject.toml` and the README returns only
|
||||||
|
the retirement notes of Requirement 4.
|
||||||
|
|
||||||
|
### Requirement 3: Deterministic behaviour is unchanged, and proven so
|
||||||
|
|
||||||
|
**User Story:** As the maintainer, I want evidence, not assurance, that removing HarnessX changed nothing a
|
||||||
|
deterministic run produces.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. Before any deletion, a baseline is recorded without the extra installed: the full host suite, and the outputs
|
||||||
|
of `ousast regress` and one `ousast benchmark` replay on the development pins, stored under
|
||||||
|
`benchmarks/measurements/`.
|
||||||
|
2. After the removal the same commands produce byte-identical outputs (timestamps and durations excluded, and
|
||||||
|
the exclusion list is written down); any difference blocks the merge.
|
||||||
|
3. A config file that still carries a `[harnessx]` section loads with one warning that names the plane, and the
|
||||||
|
section is otherwise ignored.
|
||||||
|
|
||||||
|
### Requirement 4: Documentation and specs tell the new story
|
||||||
|
|
||||||
|
**User Story:** As a reader of the README or a spec, I want to learn that the agentic plane is ax, not HarnessX.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. The README sections "Configuring the HarnessX agentic plane" and "The HarnessX self-improving cycle" are
|
||||||
|
replaced by a short section on the ax plane that points at `ops/ax/README.md`; `docs/examples.md` section 6 and
|
||||||
|
`docs/threat-model.md` (egress and spend caps) describe the plane's egress policy and budgets instead.
|
||||||
|
2. Specs that name HarnessX (`harnessx-self-improving-rulesets`, `learning-harness`, `three-stage-scan`,
|
||||||
|
`openrouter-sast-harness`) and `.kiro/steering/overview.md` get a dated retirement note at the top; their
|
||||||
|
history is not rewritten.
|
||||||
|
3. Maintainer-only Kiro tooling stays out of user docs (standing rule).
|
||||||
|
|
||||||
|
### Requirement 5: The removal lands in one reviewable sequence
|
||||||
|
|
||||||
|
**User Story:** As the maintainer, I want to review the removal in steps that each leave main green.
|
||||||
|
|
||||||
|
#### Acceptance Criteria
|
||||||
|
|
||||||
|
1. Order: baseline (3.1) -> capabilities moved or retired (1) -> code and packaging removed (2) -> equality proof
|
||||||
|
(3.2) -> docs and specs (4). Each step is a commit gated on the full suite's own exit code, ruff and mypy.
|
||||||
|
2. No step deletes code whose replacement (Requirement 1) has not landed and passed its fixture test.
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
"feature_name": "harnessx-removal",
|
||||||
|
"created_at": "2026-09-30T08:00:00Z",
|
||||||
|
"updated_at": "2026-09-30T08:00:00Z",
|
||||||
|
"language": "en",
|
||||||
|
"phase": "requirements-generated",
|
||||||
|
"approvals": {
|
||||||
|
"requirements": {"generated": true, "approved": false},
|
||||||
|
"design": {"generated": false, "approved": false},
|
||||||
|
"tasks": {"generated": false, "approved": false}
|
||||||
|
},
|
||||||
|
"ready_for_implementation": false
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user