docs(harnessx-removal): the plane replaces HarnessX in README, docs and specs (task 8)

README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.

docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.

Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 12:45:29 +02:00
co-authored by Claude Opus 5.5
parent 800204de87
commit f4bd984fda
18 changed files with 151 additions and 74 deletions
+2 -2
View File
@@ -1,9 +1,9 @@
{
"feature_name": "harnessx-removal",
"created_at": "2026-09-30T08:00:00Z",
"updated_at": "2026-09-30T11:10:00Z",
"updated_at": "2026-09-30T16:00:00Z",
"language": "en",
"phase": "implementation",
"phase": "implementation-complete",
"approvals": {
"requirements": {
"generated": true,
+1 -1
View File
@@ -55,7 +55,7 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
exclusion list blocks the merge.
- _Requirements: 3.2_
- [ ] 8. Documentation and specs
- [x] 8. Documentation and specs
- README ax-plane section pointing at `ops/ax/README.md`; `docs/examples.md` §6 and `docs/threat-model.md`
(egress policy, budgets); dated retirement notes in the four specs and `.kiro/steering/overview.md`; release
note naming the silent loss of the second hunter for users who set `[models] hunter`.
@@ -1,5 +1,8 @@
# Design Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
@@ -1,5 +1,8 @@
# Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
+3
View File
@@ -1,5 +1,8 @@
# Brief: learning-harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem
The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works.
+3
View File
@@ -1,5 +1,8 @@
# Design Document: learning-harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
@@ -1,5 +1,8 @@
# Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`.
@@ -1,5 +1,8 @@
# Design: OpenRouter SAST Harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Architecture Summary
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
@@ -1,5 +1,8 @@
# Requirements: OpenRouter SAST Harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
+3
View File
@@ -1,5 +1,8 @@
# Brief: three-stage-scan
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
+3
View File
@@ -1,5 +1,8 @@
# Design Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
@@ -1,5 +1,8 @@
# Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
+3
View File
@@ -174,4 +174,7 @@ function-level pairs could not see any of them, and a repository showed all nine
| `learning-harness` | implementation | 30 done, 3 open |
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
+46 -42
View File
@@ -215,17 +215,15 @@ ranker → hunter → verifier → mapping loop provides (critical/high severity
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
an explicit high-assurance request), fusion runs two independent panels that
steel-man the vulnerability and false-positive cases, vote, and a decider issues
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes,
model IDs, and degradations disclosed. It runs automatically on triggered findings
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes
and degradations disclosed. It runs automatically on triggered findings
in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest.
Deterministic by default (`fusion.py`); set `[fusion] panel_model` to route the
panels through the configured provider:
Fusion is deterministic (`fusion.py`); independent LLM agreement on a candidate
is the plane's `agree` task (see [The agentic plane (ax)](#the-agentic-plane-ax)):
```toml
[fusion]
enabled = true # default; runs on triggered findings in standard mode
panel_model = "gpt-4o" # optional: LLM panels via the [harnessx] provider
decider_model = "gpt-4o" # optional: disclosed in the decision's model IDs
high_assurance = false # true forces fusion on every finding
```
@@ -290,44 +288,37 @@ addressed, it stays closed because the recall/precision gate runs in CI.
`external_baseline_deltas.json` additionally shows where another tool found a
vulnerability OpenUltraSAST missed (or vice-versa).
## Configuring the HarnessX agentic plane
## The agentic plane (ax)
The LLM hunter pool and the llm-judge verifier are optional. The core install is
zero-dependency and `--mode quick` never calls a model. To enable the agentic
plane, install the extra, name the models, and pick a provider:
The core install is zero-dependency and `--mode quick` never calls a model. Model
work that spans a whole repository runs on the **agentic plane**: a Run manifest
of tasks executed on [ax](ops/ax/README.md) (Kubernetes with Agent Substrate), one
isolated actor per task. What runs there today: `repo-facts`, two independent
`verify` passes, `agree` with a 2-of-3 tie-break on disputed candidates, and
`remember`, which writes each run's per-candidate rows into the plane memory.
```bash
uv sync --extra harnessx # SHA-pinned optional dependency
export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY, matching the provider
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
uv run ousast plane status <run> # per-task status and token attribution
uv run ousast plane remember <run> # ingest the run's memory rows (OUSAST_MEMORY)
```
```toml
# openultrasast.toml
[models]
hunter = "gpt-4o" # LLM hunter pool (empty -> deterministic hunter)
verifier = "gpt-4o" # llm-judge verifier (empty -> structural verifier)
Every task binds its own Model and its own budget (`usd`, `calls`), and a task
that reaches its ceiling stops as `unfinished` instead of overspending. Egress is
deny-by-default per task; the provider key travels only in the task's start
request and is never written to an artifact. Bring-up, the egress policy and the
host requirements are in [ops/ax/README.md](ops/ax/README.md).
[harnessx]
provider = "openai" # "anthropic" (default) | "openai" | "litellm"
max_cost_usd = 2.0 # per-task spend cap
token_threshold = 120000 # per-task token budget
```
In a local `ousast scan --mode standard`, `[models] hunter` still enables the
MAP-stage tool hunter; verification and fusion are deterministic.
The provider selects the HarnessX backend for **both** the hunter and the
verifier; each reads its own standard key env var (`openai` → `OPENAI_API_KEY`,
`anthropic` → `ANTHROPIC_API_KEY`). The HarnessX path activates only when the
extra is installed, a model is configured, and the mode is `standard` — all three
gated through one capability seam (`harness_ext.has_harnessx`). If any condition
fails, the scan transparently falls back to the deterministic hunter / structural
verifier and records a `degradations` entry in `manifest.json`, so a run is never
silently downgraded:
> The earlier optional agentic extra, HarnessX, was retired 2026-09-30, and a
> leftover `[harnessx]` section (retired 2026-09-30) is ignored with one warning.
> `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with
> a message naming the plane replacement. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
```bash
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = fell back
```
## The HarnessX self-improving cycle
## The self-improving cycle
Each scan is a composed harness of typed processors with read/write **state
contracts**. Strict mode fails the scan on a violation; warn mode records a
@@ -391,11 +382,24 @@ it — and the same gate runs in CI (`python -m openultrasast.gate`), so a loop
result that breached it could never merge. Covered by `tests/test_improve.py` and
`tests/test_cli_improve.py`.
This is the deterministic substrate of the self-improvement loop. When the
optional `openultrasast[harnessx]` extra is present, the HarnessX `MetaAgent`
becomes the richer *proposer* that plugs into this same validated, gated
machinery — the safety contract (bounded levers, replay/novelty/journal gates,
hard acceptance gate, byte-for-byte revert) is identical either way.
This is the deterministic substrate of the self-improvement loop. The **plane
memory** is the richer proposer that plugs into it: with `--memory`, `ousast
improve` also reads the rows that plane runs stored per repository and pin, and two
deterministic rules turn them into rule-status proposals: a rule whose alerts are
repeatedly false across repositories is shadowed, and a shadow rule that keeps
hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs,
from the gated manifest's own cases and from the qualifying population are dropped
before any rule sees them, so the loop never learns from the cases it is judged on. Each
proposal records the memory rows it came from (`memory_proposals.jsonl` next to the
journal), and it goes through the same validator and gate as every other edit.
```bash
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
The whole loop (alerts, measure, propose, improve) can also run as a plane Run
(generated by `ousast plane workspaces --validation-set ... --loop`), with its own tasks, Models
and budgets; see [ops/ax/README.md](ops/ax/README.md).
Beyond these loops, the `openultrasast-triage` skill lets an agent adjust
prompt constraints, retrieval filters, skill routing and benchmark-miss triage.
@@ -407,7 +411,7 @@ directly once the per-family detectors land.
OpenUltraSAST treats scanned code as **untrusted input and never executes it** (quick
and standard modes). Scan artifacts are scrubbed of credentials before they are written
(`[hardening] redact_secrets`, on by default), agentic spend and output size are bounded
(`[harnessx] max_cost_usd`/`token_threshold`, `[hardening] max_findings`), provider calls
(per-task `usd`/`calls` budgets on the plane, `[hardening] max_findings`), provider calls
retry transient failures with backoff, and missing capabilities degrade visibly in the
manifest rather than silently. See [docs/threat-model.md](docs/threat-model.md) for the
full trust boundaries and sandbox limits, and [docs/examples.md](docs/examples.md) for
+31
View File
@@ -1,3 +1,34 @@
# Unreleased
## HarnessX removed; the ax plane is the agentic path (2026-09-30)
HarnessX, the optional agentic extra (`openultrasast[harnessx]`), is removed with its code,
packaging extra and lock entries (retired 2026-09-30). Agentic work runs on the ax plane:
`ousast plane run|status|doctor|remember`, with one Model and one `usd`/`calls` budget per
task, deny-by-default egress per task, and the provider key sent only in the task's start
request. See [ops/ax/README.md](ops/ax/README.md). Deterministic outputs (pair replay,
quick benchmark, standard scan, `improve --dry-run`) are byte-identical before and after.
Retired config keys (retired 2026-09-30):
- `[models] verifier` fails with `RetiredConfigError` (the CLI prints it and exits 2): LLM
verification runs on the plane (`verify` + `agree`). Remove the key.
- `[fusion] panel_model` and `[fusion] decider_model` fail the same way: fusion is
deterministic; independent LLM agreement is the plane's `agree` task. Remove the keys.
- A `[harnessx]` section (retired 2026-09-30) loads with one warning naming the plane and is
otherwise ignored.
**Silent change for `[models] hunter`.** If you set `[models] hunter` and had the extra
installed, a standard scan ran two LLM hunters: the MAP-stage tool hunter and the
HarnessX hunter pool (findings with ids `hx-hunter:<path>:<line>`). The key is still
valid and still enables the tool hunter, so nothing fails and nothing warns, but the
second hunter's findings no longer appear. Repository-wide LLM hunting runs on the plane
(`verify` passes and `agree`).
The LLM judge, the LLM fusion panels and the in-loop hosting of deterministic stages are
retired without a local replacement. `ousast improve --memory` now proposes rule-status
edits from the plane memory, through the unchanged validator and gate.
# v1.2.0-alpha.1
Experimental pre-push security safety net for AI-accelerated development. Python package
+18 -20
View File
@@ -57,30 +57,28 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
# which the next scan/benchmark of that target loads automatically.
```
## 6. Standard mode with the HarnessX agentic plane (OpenAI)
## 6. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
read from the environment (or `.env`) by `ousast` and sent only in each task's start
request.
```bash
uv sync --extra harnessx
export OPENAI_API_KEY=sk-...
cat > openultrasast.toml <<'TOML'
[models]
hunter = "gpt-4o"
verifier = "gpt-4o"
[harnessx]
provider = "openai"
max_cost_usd = 2.0
token_threshold = 120000
[fusion]
panel_model = "gpt-4o" # two-panel adjudication on triggered findings
[hardening]
redact_secrets = true
max_findings = 0
TOML
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = it fell back
jq '.fusion' "$run/manifest.json" # per-finding dispositions
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case
uv run ousast plane status validation-46 # per-task status and token attribution
uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
# Feed the stored rows back into the ruleset loop (same validator and gate)
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. A local `ousast scan --mode standard` stays deterministic apart from the
optional `[models] hunter` tool hunter.
## 7. Drive it from an MCP client (OpenCode / IDE)
```jsonc
+20 -6
View File
@@ -15,9 +15,12 @@ it does not, and the controls that keep a scan safe to run in CI and safe to sha
flip rule *status* and tune score constants but can never edit pattern text or the
authoritative 0–5 severity. See the self-improving cycle in the README.
- **Provider credentials are the operator's.** The zero-dependency core makes no network
calls. The optional HarnessX agentic plane sends source excerpts to the configured LLM
provider only when a model is set in `[models]`; keys are read from the provider's
standard env var (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY`), never written to artifacts.
calls. A local scan calls a model only for the optional `[models] hunter` tool hunter
in standard mode. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
the variable a task's Model names (`secretKey.key`) from the operator's environment or
`.env` and sends the key only in that task's start request, through Agent Substrate's
router to that task's actor. It is not baked into an image or task template, and it is
never logged or written to artifacts.
## Controls
@@ -28,10 +31,21 @@ tokens, URL-embedded credentials, PEM private keys, and `key = value` assignment
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
written. On by default; disable with `[hardening] redact_secrets = false`.
### Plane egress
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the
task's bound Workspaces and to the hosts its bound Model declares
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS
interception. The policy is deleted with the actor.
### Cost & CI budgets
- **Agentic spend** is bounded per task by `[harnessx] max_cost_usd` and
`token_threshold` (enforced inside the HarnessX run loop) and by the per-tier hunter
budgets.
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The
metered client refuses the next call once either ceiling is reached (the task is
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model
without prices is refused rather than metered as zero, and an account or
authentication error fails the task and starts nothing further. The standard-mode tool
hunter is bounded by the per-tier hunter budgets.
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
severity-ordered and disclosed as a `budget` degradation in the manifest.
-3
View File
@@ -11,8 +11,6 @@ from __future__ import annotations
import re
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1]
_NAME = "harness" + "x"
_PATTERN = re.compile(_NAME, re.IGNORECASE)
@@ -48,7 +46,6 @@ def test_the_retirement_notes_exist() -> None:
assert _PATTERN.search(config) and _TAG.search(config)
@pytest.mark.xfail(strict=True, reason="the README's plane sections are rewritten in task 8; drop this mark there")
def test_the_readme_names_the_removed_plane_only_in_its_retirement_note() -> None:
offenders = _untagged([ROOT / "README.md"])
assert offenders == [], "\n".join(offenders)