mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(harnessx-removal): the plane replaces HarnessX in README, docs and specs (task 8)
README: "Configuring the HarnessX agentic plane" becomes "The agentic plane (ax)" (ousast plane run|status|doctor|remember, per-task Model, budget and egress, pointer to ops/ax/README.md, one tagged retirement line); "The HarnessX self-improving cycle" becomes "The self-improving cycle", and its MetaAgent paragraph is replaced by the memory proposer (`ousast improve --memory`, provenance, train-on-test guard). The fusion panel_model/decider_model lines and the [harnessx] spend caps are gone. The README reference test now passes without its strict xfail. docs/examples.md section 6 runs the plane; docs/threat-model.md describes the per-task EgressPolicy, the usd/calls budgets and the start-request credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX removed, the retired keys and their errors, the silent loss of the second hunter for users who set [models] hunter, and the plane as the agentic path. Dated retirement notes (2026-09-30) head the requirements, design and brief of harnessx-self-improving-rulesets, learning-harness, three-stage-scan and openrouter-sast-harness, and follow the spec table in steering/overview.md; history is unchanged. All tasks are checked, so spec.json moves to implementation-complete. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
800204de87
commit
f4bd984fda
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "harnessx-removal",
|
||||
"created_at": "2026-09-30T08:00:00Z",
|
||||
"updated_at": "2026-09-30T11:10:00Z",
|
||||
"updated_at": "2026-09-30T16:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -55,7 +55,7 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
|
||||
exclusion list blocks the merge.
|
||||
- _Requirements: 3.2_
|
||||
|
||||
- [ ] 8. Documentation and specs
|
||||
- [x] 8. Documentation and specs
|
||||
- README ax-plane section pointing at `ops/ax/README.md`; `docs/examples.md` §6 and `docs/threat-model.md`
|
||||
(egress policy, budgets); dated retirement notes in the four specs and `.kiro/steering/overview.md`; release
|
||||
note naming the silent loss of the second hunter for users who set `[models] hunter`.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Brief: learning-harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Problem
|
||||
|
||||
The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design Document: learning-harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design: OpenRouter SAST Harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements: OpenRouter SAST Harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Brief: three-stage-scan
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Problem
|
||||
|
||||
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
|
||||
|
||||
@@ -174,4 +174,7 @@ function-level pairs could not see any of them, and a repository showed all nine
|
||||
| `learning-harness` | implementation | 30 done, 3 open |
|
||||
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
|
||||
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
|
||||
|
||||
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
|
||||
|
||||
Reference in New Issue
Block a user