mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
docs(harnessx-removal): the plane replaces HarnessX in README, docs and specs (task 8)
README: "Configuring the HarnessX agentic plane" becomes "The agentic plane (ax)" (ousast plane run|status|doctor|remember, per-task Model, budget and egress, pointer to ops/ax/README.md, one tagged retirement line); "The HarnessX self-improving cycle" becomes "The self-improving cycle", and its MetaAgent paragraph is replaced by the memory proposer (`ousast improve --memory`, provenance, train-on-test guard). The fusion panel_model/decider_model lines and the [harnessx] spend caps are gone. The README reference test now passes without its strict xfail. docs/examples.md section 6 runs the plane; docs/threat-model.md describes the per-task EgressPolicy, the usd/calls budgets and the start-request credential path. RELEASE_NOTES.md gets an Unreleased entry: HarnessX removed, the retired keys and their errors, the silent loss of the second hunter for users who set [models] hunter, and the plane as the agentic path. Dated retirement notes (2026-09-30) head the requirements, design and brief of harnessx-self-improving-rulesets, learning-harness, three-stage-scan and openrouter-sast-harness, and follow the spec table in steering/overview.md; history is unchanged. All tasks are checked, so spec.json moves to implementation-complete. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
800204de87
commit
f4bd984fda
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "harnessx-removal",
|
||||
"created_at": "2026-09-30T08:00:00Z",
|
||||
"updated_at": "2026-09-30T11:10:00Z",
|
||||
"updated_at": "2026-09-30T16:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -55,7 +55,7 @@ and passed its fixture test. Every task is a commit gated on the full suite's ow
|
||||
exclusion list blocks the merge.
|
||||
- _Requirements: 3.2_
|
||||
|
||||
- [ ] 8. Documentation and specs
|
||||
- [x] 8. Documentation and specs
|
||||
- README ax-plane section pointing at `ops/ax/README.md`; `docs/examples.md` §6 and `docs/threat-model.md`
|
||||
(egress policy, budgets); dated retirement notes in the four specs and `.kiro/steering/overview.md`; release
|
||||
note naming the silent loss of the second hunter for users who set `[models] hunter`.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Brief: learning-harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Problem
|
||||
|
||||
The tool is a hand-authored static analyzer with a learning loop bolted on the side, and the loop cannot learn. Three deliberate choices cap it: closed vocabularies (sinks, sources, obligation kinds) that grow only by writing a new spec; a corpus that is the only teacher and a shape that is the only lesson; and a ban on retaining anything the model discovers. Results have ceilings by construction, the loop trains on its own test set, and there is no failure analysis feeding the next round. Detection logic is ~16% of the source and the other 84% cannot say whether the 16% works.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design Document: learning-harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose**: Turn OpenUltraSAST from a hand-authored analyzer with a leaking improvement loop into a learning harness: an auto-classifier routes every pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved one change per round from structured failure facts; a class-aware pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; the split is enforced by construction.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST's detection numbers cannot be trusted and cannot be improved. The static machinery is treated as the detector while the model with tools already outperforms it (12/14 versus 5/14 pair-correct on the vibe-py holdout once scoring is keyed on class and function), the pair scorer keys detection on text tokens rather than on a vulnerability class, the improve loop learns from its own holdout pairs, and no change is attributed to the class it targets, so an improvement in one class silently degrades another. This feature turns the tool into a learning harness: an auto-classifier assigns every labeled pair, finding and code region to one of ten verifier-aligned families; one detector configuration per family is evolved round by round from structured failure facts; a class-aware, pair-wise scorer with unscorable denominators and a measured per-family noise floor decides acceptance; only verifier-confirmed claims are reported above suspicion; and the train/holdout split is enforced by construction. Context, measurements and sources: `brief.md`, `research.md`, `benchmarks/measurements/2026-09-06-*`.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design: OpenRouter SAST Harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
OpenUltraSAST is a harness-oriented SAST system, not a monolithic scanner. The core runtime composes processors, model roles, retrieval, tools, sandbox policy, traces, verification gates, and reporting into a reproducible scan harness.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements: OpenRouter SAST Harness
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
OpenUltraSAST shall be an independent OpenCode-driven security harness for finding, validating, reporting, and optionally fixing source-code vulnerabilities at scale. It shall use OpenRouter-selected chat models for reasoning, OpenRouter embeddings for semantic retrieval, Docker for isolated analysis, and an explicit evidence ladder to prevent AI suspicion from being reported as verified security impact.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Brief: three-stage-scan
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Problem
|
||||
|
||||
AppSec engineers and DevOps teams running OpenUltraSAST in CI get a regex static scan scored against cheat-sheet fixtures. They cannot tell what actually exists in a real repository, where complexity concentrates, how to aim tests, or whether a finding is worth a CI failure. The previous HarnessX spec added governance around those regexes (CWE policy, score, shadow status) without making the hunter evolutionary or the evaluation realistic.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Design Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose.** This feature turns an OpenUltraSAST scan into three operator questions: what exists (static inventory), where problems are most likely and how to aim tests (complexity map), and whether a finding is worth fixing (isolated regression). It is a follow-up to the HarnessX governance spec, not a replacement of CWE policy, scoring, or rules-as-data.
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
# Requirements Document
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax
|
||||
> (`ai-service-plane`), and evolutionary proposals come from the plane memory. History below is unchanged.
|
||||
|
||||
## Introduction
|
||||
|
||||
OpenUltraSAST today answers one question well: which regexes fired. It does not answer the three questions a rugged DevOps CI workflow actually asks: what exists in this repository, where problems are most likely and how to aim tests, and whether a finding is worth fixing. This feature turns the scan into those three stages. Stage 1 is a static inventory of what exists. Stage 2 is a complexity map of likely problem areas plus test-tuning hints. Stage 3 is isolated regression that proves triggerability before CI treats a finding as blocking.
|
||||
|
||||
@@ -174,4 +174,7 @@ function-level pairs could not see any of them, and a repository showed all nine
|
||||
| `learning-harness` | implementation | 30 done, 3 open |
|
||||
| six others | complete | `pair-corpus-honesty`, `corpus-seeded-mechanisms`, `propose-adjudicate-prove`, `three-stage-scan`, `real-world-vfc-slice`, `harnessx-self-improving-rulesets`, `tree-sitter-overlay-extra` |
|
||||
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
|
||||
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
|
||||
|
||||
97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
|
||||
|
||||
@@ -215,17 +215,15 @@ ranker → hunter → verifier → mapping loop provides (critical/high severity
|
||||
verifier disagreement, conflicting static vs semantic evidence, risky fixes, or
|
||||
an explicit high-assurance request), fusion runs two independent panels that
|
||||
steel-man the vulnerability and false-positive cases, vote, and a decider issues
|
||||
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes,
|
||||
model IDs, and degradations disclosed. It runs automatically on triggered findings
|
||||
the disposition (accepted / rejected / mitigated / deferred / blocked) with votes
|
||||
and degradations disclosed. It runs automatically on triggered findings
|
||||
in `--mode standard`, writing `fusion.json` and a `fusion` block in the manifest.
|
||||
Deterministic by default (`fusion.py`); set `[fusion] panel_model` to route the
|
||||
panels through the configured provider:
|
||||
Fusion is deterministic (`fusion.py`); independent LLM agreement on a candidate
|
||||
is the plane's `agree` task (see [The agentic plane (ax)](#the-agentic-plane-ax)):
|
||||
|
||||
```toml
|
||||
[fusion]
|
||||
enabled = true # default; runs on triggered findings in standard mode
|
||||
panel_model = "gpt-4o" # optional: LLM panels via the [harnessx] provider
|
||||
decider_model = "gpt-4o" # optional: disclosed in the decision's model IDs
|
||||
high_assurance = false # true forces fusion on every finding
|
||||
```
|
||||
|
||||
@@ -290,44 +288,37 @@ addressed, it stays closed because the recall/precision gate runs in CI.
|
||||
`external_baseline_deltas.json` additionally shows where another tool found a
|
||||
vulnerability OpenUltraSAST missed (or vice-versa).
|
||||
|
||||
## Configuring the HarnessX agentic plane
|
||||
## The agentic plane (ax)
|
||||
|
||||
The LLM hunter pool and the llm-judge verifier are optional. The core install is
|
||||
zero-dependency and `--mode quick` never calls a model. To enable the agentic
|
||||
plane, install the extra, name the models, and pick a provider:
|
||||
The core install is zero-dependency and `--mode quick` never calls a model. Model
|
||||
work that spans a whole repository runs on the **agentic plane**: a Run manifest
|
||||
of tasks executed on [ax](ops/ax/README.md) (Kubernetes with Agent Substrate), one
|
||||
isolated actor per task. What runs there today: `repo-facts`, two independent
|
||||
`verify` passes, `agree` with a 2-of-3 tie-break on disputed candidates, and
|
||||
`remember`, which writes each run's per-candidate rows into the plane memory.
|
||||
|
||||
```bash
|
||||
uv sync --extra harnessx # SHA-pinned optional dependency
|
||||
export OPENAI_API_KEY=sk-... # or ANTHROPIC_API_KEY, matching the provider
|
||||
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
|
||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
|
||||
uv run ousast plane status <run> # per-task status and token attribution
|
||||
uv run ousast plane remember <run> # ingest the run's memory rows (OUSAST_MEMORY)
|
||||
```
|
||||
|
||||
```toml
|
||||
# openultrasast.toml
|
||||
[models]
|
||||
hunter = "gpt-4o" # LLM hunter pool (empty -> deterministic hunter)
|
||||
verifier = "gpt-4o" # llm-judge verifier (empty -> structural verifier)
|
||||
Every task binds its own Model and its own budget (`usd`, `calls`), and a task
|
||||
that reaches its ceiling stops as `unfinished` instead of overspending. Egress is
|
||||
deny-by-default per task; the provider key travels only in the task's start
|
||||
request and is never written to an artifact. Bring-up, the egress policy and the
|
||||
host requirements are in [ops/ax/README.md](ops/ax/README.md).
|
||||
|
||||
[harnessx]
|
||||
provider = "openai" # "anthropic" (default) | "openai" | "litellm"
|
||||
max_cost_usd = 2.0 # per-task spend cap
|
||||
token_threshold = 120000 # per-task token budget
|
||||
```
|
||||
In a local `ousast scan --mode standard`, `[models] hunter` still enables the
|
||||
MAP-stage tool hunter; verification and fusion are deterministic.
|
||||
|
||||
The provider selects the HarnessX backend for **both** the hunter and the
|
||||
verifier; each reads its own standard key env var (`openai` → `OPENAI_API_KEY`,
|
||||
`anthropic` → `ANTHROPIC_API_KEY`). The HarnessX path activates only when the
|
||||
extra is installed, a model is configured, and the mode is `standard` — all three
|
||||
gated through one capability seam (`harness_ext.has_harnessx`). If any condition
|
||||
fails, the scan transparently falls back to the deterministic hunter / structural
|
||||
verifier and records a `degradations` entry in `manifest.json`, so a run is never
|
||||
silently downgraded:
|
||||
> The earlier optional agentic extra, HarnessX, was retired 2026-09-30, and a
|
||||
> leftover `[harnessx]` section (retired 2026-09-30) is ignored with one warning.
|
||||
> `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail with
|
||||
> a message naming the plane replacement. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
|
||||
```bash
|
||||
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = fell back
|
||||
```
|
||||
|
||||
## The HarnessX self-improving cycle
|
||||
## The self-improving cycle
|
||||
|
||||
Each scan is a composed harness of typed processors with read/write **state
|
||||
contracts**. Strict mode fails the scan on a violation; warn mode records a
|
||||
@@ -391,11 +382,24 @@ it — and the same gate runs in CI (`python -m openultrasast.gate`), so a loop
|
||||
result that breached it could never merge. Covered by `tests/test_improve.py` and
|
||||
`tests/test_cli_improve.py`.
|
||||
|
||||
This is the deterministic substrate of the self-improvement loop. When the
|
||||
optional `openultrasast[harnessx]` extra is present, the HarnessX `MetaAgent`
|
||||
becomes the richer *proposer* that plugs into this same validated, gated
|
||||
machinery — the safety contract (bounded levers, replay/novelty/journal gates,
|
||||
hard acceptance gate, byte-for-byte revert) is identical either way.
|
||||
This is the deterministic substrate of the self-improvement loop. The **plane
|
||||
memory** is the richer proposer that plugs into it: with `--memory`, `ousast
|
||||
improve` also reads the rows that plane runs stored per repository and pin, and two
|
||||
deterministic rules turn them into rule-status proposals: a rule whose alerts are
|
||||
repeatedly false across repositories is shadowed, and a shadow rule that keeps
|
||||
hitting agreed, declared vulnerable sites is re-enabled. Rows from holdout pairs,
|
||||
from the gated manifest's own cases and from the qualifying population are dropped
|
||||
before any rule sees them, so the loop never learns from the cases it is judged on. Each
|
||||
proposal records the memory rows it came from (`memory_proposals.jsonl` next to the
|
||||
journal), and it goes through the same validator and gate as every other edit.
|
||||
|
||||
```bash
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||
```
|
||||
|
||||
The whole loop (alerts, measure, propose, improve) can also run as a plane Run
|
||||
(generated by `ousast plane workspaces --validation-set ... --loop`), with its own tasks, Models
|
||||
and budgets; see [ops/ax/README.md](ops/ax/README.md).
|
||||
|
||||
Beyond these loops, the `openultrasast-triage` skill lets an agent adjust
|
||||
prompt constraints, retrieval filters, skill routing and benchmark-miss triage.
|
||||
@@ -407,7 +411,7 @@ directly once the per-family detectors land.
|
||||
OpenUltraSAST treats scanned code as **untrusted input and never executes it** (quick
|
||||
and standard modes). Scan artifacts are scrubbed of credentials before they are written
|
||||
(`[hardening] redact_secrets`, on by default), agentic spend and output size are bounded
|
||||
(`[harnessx] max_cost_usd`/`token_threshold`, `[hardening] max_findings`), provider calls
|
||||
(per-task `usd`/`calls` budgets on the plane, `[hardening] max_findings`), provider calls
|
||||
retry transient failures with backoff, and missing capabilities degrade visibly in the
|
||||
manifest rather than silently. See [docs/threat-model.md](docs/threat-model.md) for the
|
||||
full trust boundaries and sandbox limits, and [docs/examples.md](docs/examples.md) for
|
||||
|
||||
@@ -1,3 +1,34 @@
|
||||
# Unreleased
|
||||
|
||||
## HarnessX removed; the ax plane is the agentic path (2026-09-30)
|
||||
|
||||
HarnessX, the optional agentic extra (`openultrasast[harnessx]`), is removed with its code,
|
||||
packaging extra and lock entries (retired 2026-09-30). Agentic work runs on the ax plane:
|
||||
`ousast plane run|status|doctor|remember`, with one Model and one `usd`/`calls` budget per
|
||||
task, deny-by-default egress per task, and the provider key sent only in the task's start
|
||||
request. See [ops/ax/README.md](ops/ax/README.md). Deterministic outputs (pair replay,
|
||||
quick benchmark, standard scan, `improve --dry-run`) are byte-identical before and after.
|
||||
|
||||
Retired config keys (retired 2026-09-30):
|
||||
|
||||
- `[models] verifier` fails with `RetiredConfigError` (the CLI prints it and exits 2): LLM
|
||||
verification runs on the plane (`verify` + `agree`). Remove the key.
|
||||
- `[fusion] panel_model` and `[fusion] decider_model` fail the same way: fusion is
|
||||
deterministic; independent LLM agreement is the plane's `agree` task. Remove the keys.
|
||||
- A `[harnessx]` section (retired 2026-09-30) loads with one warning naming the plane and is
|
||||
otherwise ignored.
|
||||
|
||||
**Silent change for `[models] hunter`.** If you set `[models] hunter` and had the extra
|
||||
installed, a standard scan ran two LLM hunters: the MAP-stage tool hunter and the
|
||||
HarnessX hunter pool (findings with ids `hx-hunter:<path>:<line>`). The key is still
|
||||
valid and still enables the tool hunter, so nothing fails and nothing warns, but the
|
||||
second hunter's findings no longer appear. Repository-wide LLM hunting runs on the plane
|
||||
(`verify` passes and `agree`).
|
||||
|
||||
The LLM judge, the LLM fusion panels and the in-loop hosting of deterministic stages are
|
||||
retired without a local replacement. `ousast improve --memory` now proposes rule-status
|
||||
edits from the plane memory, through the unchanged validator and gate.
|
||||
|
||||
# v1.2.0-alpha.1
|
||||
|
||||
Experimental pre-push security safety net for AI-accelerated development. Python package
|
||||
|
||||
+18
-20
@@ -57,30 +57,28 @@ uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml
|
||||
# which the next scan/benchmark of that target loads automatically.
|
||||
```
|
||||
|
||||
## 6. Standard mode with the HarnessX agentic plane (OpenAI)
|
||||
## 6. Running the agentic plane
|
||||
|
||||
Model work over whole repositories runs as a Run manifest on ax, one isolated actor
|
||||
per task (bring-up: [ops/ax/README.md](../ops/ax/README.md)). The provider key is
|
||||
read from the environment (or `.env`) by `ousast` and sent only in each task's start
|
||||
request.
|
||||
|
||||
```bash
|
||||
uv sync --extra harnessx
|
||||
export OPENAI_API_KEY=sk-...
|
||||
cat > openultrasast.toml <<'TOML'
|
||||
[models]
|
||||
hunter = "gpt-4o"
|
||||
verifier = "gpt-4o"
|
||||
[harnessx]
|
||||
provider = "openai"
|
||||
max_cost_usd = 2.0
|
||||
token_threshold = 120000
|
||||
[fusion]
|
||||
panel_model = "gpt-4o" # two-panel adjudication on triggered findings
|
||||
[hardening]
|
||||
redact_secrets = true
|
||||
max_findings = 0
|
||||
TOML
|
||||
uv run ousast scan /path/to/code --mode standard --config openultrasast.toml
|
||||
jq '.degradations' "$run/manifest.json" # null = HarnessX ran; entries = it fell back
|
||||
jq '.fusion' "$run/manifest.json" # per-finding dispositions
|
||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # repo-facts, verify a/b(/c), agree, remember per case
|
||||
uv run ousast plane status validation-46 # per-task status and token attribution
|
||||
uv run ousast plane remember validation-46 # re-ingest the run's memory rows (OUSAST_MEMORY)
|
||||
|
||||
# Feed the stored rows back into the ruleset loop (same validator and gate)
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||
```
|
||||
|
||||
Each task in the Run carries its own Model and `budget: {usd, calls}`; a task that
|
||||
reaches its ceiling stops as `unfinished` and resumes on a rerun with a larger
|
||||
budget. A local `ousast scan --mode standard` stays deterministic apart from the
|
||||
optional `[models] hunter` tool hunter.
|
||||
|
||||
## 7. Drive it from an MCP client (OpenCode / IDE)
|
||||
|
||||
```jsonc
|
||||
|
||||
+20
-6
@@ -15,9 +15,12 @@ it does not, and the controls that keep a scan safe to run in CI and safe to sha
|
||||
flip rule *status* and tune score constants but can never edit pattern text or the
|
||||
authoritative 0–5 severity. See the self-improving cycle in the README.
|
||||
- **Provider credentials are the operator's.** The zero-dependency core makes no network
|
||||
calls. The optional HarnessX agentic plane sends source excerpts to the configured LLM
|
||||
provider only when a model is set in `[models]`; keys are read from the provider's
|
||||
standard env var (`OPENAI_API_KEY` / `ANTHROPIC_API_KEY`), never written to artifacts.
|
||||
calls. A local scan calls a model only for the optional `[models] hunter` tool hunter
|
||||
in standard mode. Agentic work runs on the plane (`ousast plane run`): `ousast` reads
|
||||
the variable a task's Model names (`secretKey.key`) from the operator's environment or
|
||||
`.env` and sends the key only in that task's start request, through Agent Substrate's
|
||||
router to that task's actor. It is not baked into an image or task template, and it is
|
||||
never logged or written to artifacts.
|
||||
|
||||
## Controls
|
||||
|
||||
@@ -28,10 +31,21 @@ tokens, URL-embedded credentials, PEM private keys, and `key = value` assignment
|
||||
sensitive names) before traces (`trace/events.jsonl`) and the markdown report are
|
||||
written. On by default; disable with `[hardening] redact_secrets = false`.
|
||||
|
||||
### Plane egress
|
||||
Each plane task runs in its own actor behind Agent Substrate's egress gateway, which
|
||||
denies by default. The task's EgressPolicy (`src/openultrasast/plane/egress.py`) allows exactly three
|
||||
things: plain HTTP to the artifact receiver, and TLS passthrough to the Git hosts of the
|
||||
task's bound Workspaces and to the hosts its bound Model declares
|
||||
(`openultrasast.io/egress-hosts`). There are no wildcards, no IP addresses and no TLS
|
||||
interception. The policy is deleted with the actor.
|
||||
|
||||
### Cost & CI budgets
|
||||
- **Agentic spend** is bounded per task by `[harnessx] max_cost_usd` and
|
||||
`token_threshold` (enforced inside the HarnessX run loop) and by the per-tier hunter
|
||||
budgets.
|
||||
- **Agentic spend** is bounded per plane task by the Run's `budget: {usd, calls}`. The
|
||||
metered client refuses the next call once either ceiling is reached (the task is
|
||||
`unfinished` and resumes on a rerun with a larger budget). A usd budget on a Model
|
||||
without prices is refused rather than metered as zero, and an account or
|
||||
authentication error fails the task and starts nothing further. The standard-mode tool
|
||||
hunter is bounded by the per-tier hunter budgets.
|
||||
- **Output size** is bounded by `[hardening] max_findings` (0 = unlimited). Truncation is
|
||||
severity-ordered and disclosed as a `budget` degradation in the manifest.
|
||||
|
||||
|
||||
@@ -11,8 +11,6 @@ from __future__ import annotations
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
_NAME = "harness" + "x"
|
||||
_PATTERN = re.compile(_NAME, re.IGNORECASE)
|
||||
@@ -48,7 +46,6 @@ def test_the_retirement_notes_exist() -> None:
|
||||
assert _PATTERN.search(config) and _TAG.search(config)
|
||||
|
||||
|
||||
@pytest.mark.xfail(strict=True, reason="the README's plane sections are rewritten in task 8; drop this mark there")
|
||||
def test_the_readme_names_the_removed_plane_only_in_its_retirement_note() -> None:
|
||||
offenders = _untagged([ROOT / "README.md"])
|
||||
assert offenders == [], "\n".join(offenders)
|
||||
|
||||
Reference in New Issue
Block a user