docs(harnessx-removal): the plane replaces HarnessX in README, docs and specs (task 8)

README: "Configuring the HarnessX agentic plane" becomes "The agentic
plane (ax)" (ousast plane run|status|doctor|remember, per-task Model,
budget and egress, pointer to ops/ax/README.md, one tagged retirement
line); "The HarnessX self-improving cycle" becomes "The self-improving
cycle", and its MetaAgent paragraph is replaced by the memory proposer
(`ousast improve --memory`, provenance, train-on-test guard). The fusion
panel_model/decider_model lines and the [harnessx] spend caps are gone.
The README reference test now passes without its strict xfail.

docs/examples.md section 6 runs the plane; docs/threat-model.md describes
the per-task EgressPolicy, the usd/calls budgets and the start-request
credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX
removed, the retired keys and their errors, the silent loss of the
second hunter for users who set [models] hunter, and the plane as the
agentic path.

Dated retirement notes (2026-09-30) head the requirements, design and
brief of harnessx-self-improving-rulesets, learning-harness,
three-stage-scan and openrouter-sast-harness, and follow the spec table
in steering/overview.md; history is unchanged. All tasks are checked, so
spec.json moves to implementation-complete.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
Marius Ciepluch
2026-09-30 12:45:29 +02:00
co-authored by Claude Opus 5.5
parent 800204de87
commit f4bd984fda
18 changed files with 151 additions and 74 deletions
+2 -2
View File
@@ -1,9 +1,9 @@
{ {
"feature_name": "harnessx-removal", "feature_name": "harnessx-removal",
"created_at": "2026-09-30T08:00:00Z", "created_at": "2026-09-30T08:00:00Z",
"updated_at": "2026-09-30T11:10:00Z", "updated_at": "2026-09-30T16:00:00Z",
"language": "en", "language": "en",
"phase": "implementation", "phase": "implementation-complete",
"approvals": { "approvals": {
"requirements": { "requirements": {
"generated": true, "generated": true,
+1 -1
View File
@@ -55,7 +55,7 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
exclusion list blocks the merge. exclusion list blocks the merge.
- _Requirements: 3.2_ - _Requirements: 3.2_
- [ ] 8. Documentation and specs - [x] 8. Documentation and specs
- README ax-plane section pointing at `ops/ax/README.md`; `docs/examples.md` §6 and `docs/threat-model.md` - README ax-plane section pointing at `ops/ax/README.md`; `docs/examples.md` §6 and `docs/threat-model.md`
(egress policy, budgets); dated retirement notes in the four specs and `.kiro/steering/overview.md`; release (egress policy, budgets); dated retirement notes in the four specs and `.kiro/steering/overview.md`; release
note naming the silent loss of the second hunter for users who set `[models] hunter`. note naming the silent loss of the second hunter for users who set `[models] hunter`.
@@ -1,5 +1,8 @@
# Design Document # Design Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview ## Overview
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds. **Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
@@ -1,5 +1,8 @@
# Requirements Document # Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction ## Introduction
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds. OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
+3
View File
@@ -1,5 +1,8 @@
# Brief: learning-harness # Brief: learning-harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem ## Problem
The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works. The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works.
+3
View File
@@ -1,5 +1,8 @@
# Design Document: learning-harness # Design Document: learning-harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview ## Overview
**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction. **Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
@@ -1,5 +1,8 @@
# Requirements Document # Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction ## Introduction
OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`. OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`.
@@ -1,5 +1,8 @@
# Design: OpenRouter SAST Harness # Design: OpenRouter SAST Harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Architecture Summary ## Architecture Summary
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness. OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
@@ -1,5 +1,8 @@
# Requirements: OpenRouter SAST Harness # Requirements: OpenRouter SAST Harness
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview ## Overview
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact. OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
+3
View File
@@ -1,5 +1,8 @@
# Brief: three-stage-scan # Brief: three-stage-scan
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Problem ## Problem
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic. AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
+3
View File
@@ -1,5 +1,8 @@
# Design Document # Design Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Overview ## Overview
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data. **Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
@@ -1,5 +1,8 @@
# Requirements Document # Requirements Document
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
## Introduction ## Introduction
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking. OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
+3
View File
@@ -174,4 +174,7 @@ function-level pairs could not see any of them, and a repository showed all nine
| `learning-harness` | implementation | 30 done, 3 open | | `learning-harness` | implementation | 30 done, 3 open |
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` | | six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests. 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
+46 -42
View File
@@ -215,17 +215,15 @@ ranker → hunter → verifier → mapping loop provides (critical/high severity
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
an explicit high-assurance request), fusion runs two independent panels that an explicit high-assurance request), fusion runs two independent panels that
steel-man the vulnerability and false-positive cases, vote, and a decider issues steel-man the vulnerability and false-positive cases, vote, and a decider issues
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes, the disposition (accepted / rejected / mitigated / deferred / blocked) with votes
model IDs, and degradations disclosed. It runs automatically on triggered findings and degradations disclosed. It runs automatically on triggered findings
in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest. in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest.
Deterministic by default (`fusion.py`); set `[fusion] panel_model` to route the Fusion is deterministic (`fusion.py`); independent LLM agreement on a candidate
panels through the configured provider: is the plane's `agree` task (see [The agentic plane (ax)](#the-agentic-plane-ax)):
```toml ```toml
[fusion] [fusion]
enabled = true # default; runs on triggered findings in standard mode enabled = true # default; runs on triggered findings in standard mode
panel_model = "gpt-4o" # optional: LLM panels via the [harnessx] provider
decider_model = "gpt-4o" # optional: disclosed in the decision's model IDs
high_assurance = false # true forces fusion on every finding high_assurance = false # true forces fusion on every finding
``` ```
@@ -290,44 +288,37 @@ addressed, it stays closed because the recall/precision gate runs in CI.
`external_baseline_deltas.json` additionally shows where another tool found a `external_baseline_deltas.json` additionally shows where another tool found a
vulnerability OpenUltraSAST missed (or vice-versa). vulnerability OpenUltraSAST missed (or vice-versa).
## Configuring the HarnessX agentic plane ## The agentic plane (ax)
The LLM hunter pool and the llm-judge verifier are optional. The core install is The core install is zero-dependency and `--mode quick` never calls a model. Model
zero-dependency and `--mode quick` never calls a model. To enable the agentic work that spans a whole repository runs on the **agentic plane**: a Run manifest
plane, install the extra, name the models, and pick a provider: of tasks executed on [ax](ops/ax/README.md) (Kubernetes with Agent Substrate), one
isolated actor per task. What runs there today: `repo-facts`, two independent
`verify` passes, `agree` with a 2-of-3 tie-break on disputed candidates, and
`remember`, which writes each run's per-candidate rows into the plane memory.
```bash ```bash
uv sync --extra harnessx # SHA-pinned optional dependency uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY, matching the provider uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml uv run ousast plane status <run> # per-task status and token attribution
uv run ousast plane remember <run> # ingest the run's memory rows (OUSAST_MEMORY)
``` ```
```toml Every task binds its own Model and its own budget (`usd`, `calls`), and a task
# openultrasast.toml that reaches its ceiling stops as `unfinished` instead of overspending. Egress is
[models] deny-by-default per task; the provider key travels only in the task's start
hunter = "gpt-4o" # LLM hunter pool (empty -> deterministic hunter) request and is never written to an artifact. Bring-up, the egress policy and the
verifier = "gpt-4o" # llm-judge verifier (empty -> structural verifier) host requirements are in [ops/ax/README.md](ops/ax/README.md).
[harnessx] In a local `ousast scan --mode standard`, `[models] hunter` still enables the
provider = "openai" # "anthropic" (default) | "openai" | "litellm" MAP-stage tool hunter; verification and fusion are deterministic.
max_cost_usd = 2.0 # per-task spend cap
token_threshold = 120000 # per-task token budget
```
The provider selects the HarnessX backend for **both** the hunter and the > The earlier optional agentic extra, HarnessX, was retired 2026-09-30, and a
verifier; each reads its own standard key env var (`openai` → `OPENAI_API_KEY`, > leftover `[harnessx]` section (retired 2026-09-30) is ignored with one warning.
`anthropic` → `ANTHROPIC_API_KEY`). The HarnessX path activates only when the > `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with
extra is installed, a model is configured, and the mode is `standard` — all three > a message naming the plane replacement. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
gated through one capability seam (`harness_ext.has_harnessx`). If any condition
fails, the scan transparently falls back to the deterministic hunter / structural
verifier and records a `degradations` entry in `manifest.json`, so a run is never
silently downgraded:
```bash ## The self-improving cycle
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = fell back
```
## The HarnessX self-improving cycle
Each scan is a composed harness of typed processors with read/write **state Each scan is a composed harness of typed processors with read/write **state
contracts**. Strict mode fails the scan on a violation; warn mode records a contracts**. Strict mode fails the scan on a violation; warn mode records a
@@ -391,11 +382,24 @@ it — and the same gate runs in CI (`python -m openultrasast.gate`), so a loop
result that breached it could never merge. Covered by `tests/test_improve.py` and result that breached it could never merge. Covered by `tests/test_improve.py` and
`tests/test_cli_improve.py`. `tests/test_cli_improve.py`.
This is the deterministic substrate of the self-improvement loop. When the This is the deterministic substrate of the self-improvement loop. The **plane
optional `openultrasast[harnessx]` extra is present, the HarnessX `MetaAgent` memory** is the richer proposer that plugs into it: with `--memory`, `ousast
becomes the richer *proposer* that plugs into this same validated, gated improve` also reads the rows that plane runs stored per repository and pin, and two
machinery — the safety contract (bounded levers, replay/novelty/journal gates, deterministic rules turn them into rule-status proposals: a rule whose alerts are
hard acceptance gate, byte-for-byte revert) is identical either way. repeatedly false across repositories is shadowed, and a shadow rule that keeps
hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs,
from the gated manifest's own cases and from the qualifying population are dropped
before any rule sees them, so the loop never learns from the cases it is judged on. Each
proposal records the memory rows it came from (`memory_proposals.jsonl` next to the
journal), and it goes through the same validator and gate as every other edit.
```bash
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
```
The whole loop (alerts, measure, propose, improve) can also run as a plane Run
(generated by `ousast plane workspaces --validation-set ... --loop`), with its own tasks, Models
and budgets; see [ops/ax/README.md](ops/ax/README.md).
Beyond these loops, the `openultrasast-triage` skill lets an agent adjust Beyond these loops, the `openultrasast-triage` skill lets an agent adjust
prompt constraints, retrieval filters, skill routing and benchmark-miss triage. prompt constraints, retrieval filters, skill routing and benchmark-miss triage.
@@ -407,7 +411,7 @@ directly once the per-family detectors land.
OpenUltraSAST treats scanned code as **untrusted input and never executes it** (quick OpenUltraSAST treats scanned code as **untrusted input and never executes it** (quick
and standard modes). Scan artifacts are scrubbed of credentials before they are written and standard modes). Scan artifacts are scrubbed of credentials before they are written
(`[hardening] redact_secrets`, on by default), agentic spend and output size are bounded (`[hardening] redact_secrets`, on by default), agentic spend and output size are bounded
(`[harnessx] max_cost_usd`/`token_threshold`, `[hardening] max_findings`), provider calls (per-task `usd`/`calls` budgets on the plane, `[hardening] max_findings`), provider calls
retry transient failures with backoff, and missing capabilities degrade visibly in the retry transient failures with backoff, and missing capabilities degrade visibly in the
manifest rather than silently. See [docs/threat-model.md](docs/threat-model.md) for the manifest rather than silently. See [docs/threat-model.md](docs/threat-model.md) for the
full trust boundaries and sandbox limits, and [docs/examples.md](docs/examples.md) for full trust boundaries and sandbox limits, and [docs/examples.md](docs/examples.md) for
+31
View File
@@ -1,3 +1,34 @@
# Unreleased
## HarnessX removed; the ax plane is the agentic path (2026-09-30)
HarnessX, the optional agentic extra (`openultrasast[harnessx]`), is removed with its code,
packaging extra and lock entries (retired 2026-09-30). Agentic work runs on the ax plane:
`ousast plane run|status|doctor|remember`, with one Model and one `usd`/`calls` budget per
task, deny-by-default egress per task, and the provider key sent only in the task's start
request. See [ops/ax/README.md](ops/ax/README.md). Deterministic outputs (pair replay,
quick benchmark, standard scan, `improve --dry-run`) are byte-identical before and after.
Retired config keys (retired 2026-09-30):
- `[models] verifier` fails with `RetiredConfigError` (the CLI prints it and exits 2): LLM
verification runs on the plane (`verify` + `agree`). Remove the key.
- `[fusion] panel_model` and `[fusion] decider_model` fail the same way: fusion is
deterministic; independent LLM agreement is the plane's `agree` task. Remove the keys.
- A `[harnessx]` section (retired 2026-09-30) loads with one warning naming the plane and is
otherwise ignored.
**Silent change for `[models] hunter`.** If you set `[models] hunter` and had the extra
installed, a standard scan ran two LLM hunters: the MAP-stage tool hunter and the
HarnessX hunter pool (findings with ids `hx-hunter:<path>:<line>`). The key is still
valid and still enables the tool hunter, so nothing fails and nothing warns, but the
second hunter's findings no longer appear. Repository-wide LLM hunting runs on the plane
(`verify` passes and `agree`).
The LLM judge, the LLM fusion panels and the in-loop hosting of deterministic stages are
retired without a local replacement. `ousast improve --memory` now proposes rule-status
edits from the plane memory, through the unchanged validator and gate.
# v1.2.0-alpha.1 # v1.2.0-alpha.1
Experimental pre-push security safety net for AI-accelerated development. Python package Experimental pre-push security safety net for AI-accelerated development. Python package
+18 -20
View File
@@ -57,30 +57,28 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
# which the next scan/benchmark of that target loads automatically. # which the next scan/benchmark of that target loads automatically.
``` ```
## 6. Standard mode with the HarnessX agentic plane (OpenAI) ## 6. Running the agentic plane
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
read from the environment (or `.env`) by `ousast` and sent only in each task's start
request.
```bash ```bash
uv sync --extra harnessx uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
export OPENAI_API_KEY=sk-... uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case
cat > openultrasast.toml <<'TOML' uv run ousast plane status validation-46 # per-task status and token attribution
[models] uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
hunter = "gpt-4o"
verifier = "gpt-4o" # Feed the stored rows back into the ruleset loop (same validator and gate)
[harnessx] uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
provider = "openai"
max_cost_usd = 2.0
token_threshold = 120000
[fusion]
panel_model = "gpt-4o" # two-panel adjudication on triggered findings
[hardening]
redact_secrets = true
max_findings = 0
TOML
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = it fell back
jq '.fusion' "$run/manifest.json" # per-finding dispositions
``` ```
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
budget. A local `ousast scan --mode standard` stays deterministic apart from the
optional `[models] hunter` tool hunter.
## 7. Drive it from an MCP client (OpenCode / IDE) ## 7. Drive it from an MCP client (OpenCode / IDE)
```jsonc ```jsonc
+20 -6
View File
@@ -15,9 +15,12 @@ it does not, and the controls that keep a scan safe to run in CI and safe to sha
flip rule *status* and tune score constants but can never edit pattern text or the flip rule *status* and tune score constants but can never edit pattern text or the
authoritative 0–5 severity. See the self-improving cycle in the README. authoritative 0–5 severity. See the self-improving cycle in the README.
- **Provider credentials are the operator's.** The zero-dependency core makes no network - **Provider credentials are the operator's.** The zero-dependency core makes no network
calls. The optional HarnessX agentic plane sends source excerpts to the configured LLM calls. A local scan calls a model only for the optional `[models] hunter` tool hunter
provider only when a model is set in `[models]`; keys are read from the provider's in standard mode. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
standard env var (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY`), never written to artifacts. the variable a task's Model names (`secretKey.key`) from the operator's environment or
`.env` and sends the key only in that task's start request, through Agent Substrate's
router to that task's actor. It is not baked into an image or task template, and it is
never logged or written to artifacts.
## Controls ## Controls
@@ -28,10 +31,21 @@ tokens, URL-embedded credentials, PEM private keys, and `key = value` assignment
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
written. On by default; disable with `[hardening] redact_secrets = false`. written. On by default; disable with `[hardening] redact_secrets = false`.
### Plane egress
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the
task's bound Workspaces and to the hosts its bound Model declares
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS
interception. The policy is deleted with the actor.
### Cost & CI budgets ### Cost & CI budgets
- **Agentic spend** is bounded per task by `[harnessx] max_cost_usd` and - **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The
`token_threshold` (enforced inside the HarnessX run loop) and by the per-tier hunter metered client refuses the next call once either ceiling is reached (the task is
budgets. `unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model
without prices is refused rather than metered as zero, and an account or
authentication error fails the task and starts nothing further. The standard-mode tool
hunter is bounded by the per-tier hunter budgets.
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is - **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
severity-ordered and disclosed as a `budget` degradation in the manifest. severity-ordered and disclosed as a `budget` degradation in the manifest.
-3
View File
@@ -11,8 +11,6 @@ from __future__ import annotations
import re import re
from pathlib import Path from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1] ROOT = Path(__file__).resolve().parents[1]
_NAME = "harness" + "x" _NAME = "harness" + "x"
_PATTERN = re.compile(_NAME, re.IGNORECASE) _PATTERN = re.compile(_NAME, re.IGNORECASE)
@@ -48,7 +46,6 @@ def test_the_retirement_notes_exist() -> None:
assert _PATTERN.search(config) and _TAG.search(config) assert _PATTERN.search(config) and _TAG.search(config)
@pytest.mark.xfail(strict=True, reason="the README's plane sections are rewritten in task 8; drop this mark there")
def test_the_readme_names_the_removed_plane_only_in_its_retirement_note() -> None: def test_the_readme_names_the_removed_plane_only_in_its_retirement_note() -> None:
offenders = _untagged([ROOT / "README.md"]) offenders = _untagged([ROOT / "README.md"])
assert offenders == [], "\n".join(offenders) assert offenders == [], "\n".join(offenders)