docs(harnessx-removal): the plane replaces HarnessX in README, docs and specs (task 8)

README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.

docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.

Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 12:45:29 +02:00
co-authored by Claude Opus 5.5
parent 800204de87
commit f4bd984fda
18 changed files with 151 additions and 74 deletions
+2 -2
View File
@@ -1,9 +1,9 @@
{
"feature_name": "harnessx-removal",
"created_at": "2026-09-30T08:00:00Z",
"updated_at": "2026-09-30T11:10:00Z",
"updated_at": "2026-09-30T16:00:00Z",
"language": "en",
"phase": "implementation",
"phase": "implementation-complete",
"approvals": {
"requirements": {
"generated": true,
+1 -1
View File
@@ -55,7 +55,7 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
exclusion list blocks the merge.
- _Requirements: 3.2_
- [ ] 8. Documentation and specs
- [x] 8. Documentation and specs
- README ax-plane section pointing at `ops/ax/README.md`; `docs/examples.md` §6 and `docs/threat-model.md`
(egress policy, budgets); dated retirement notes in the four specs and `.kiro/steering/overview.md`; release
note naming the silent loss of the second hunter for users who set `[models] hunter`.
@@ -1,5 +1,8 @@
# Design Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
@@ -1,5 +1,8 @@
# Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
+3
View File
@@ -1,5 +1,8 @@
# Brief: learning-harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem
The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works.
+3
View File
@@ -1,5 +1,8 @@
# Design Document: learning-harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
@@ -1,5 +1,8 @@
# Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`.
@@ -1,5 +1,8 @@
# Design: OpenRouter SAST Harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Architecture Summary
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
@@ -1,5 +1,8 @@
# Requirements: OpenRouter SAST Harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
+3
View File
@@ -1,5 +1,8 @@
# Brief: three-stage-scan
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
+3
View File
@@ -1,5 +1,8 @@
# Design Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
@@ -1,5 +1,8 @@
# Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
+3
View File
@@ -174,4 +174,7 @@ function-level pairs could not see any of them, and a repository showed all nine
| `learning-harness` | implementation | 30 done, 3 open |
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.