**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds.
**Users.** OpenUltraSAST maintainers gain a staged, reversible HarnessX adoption and an automatic, gated rule/policy tuning loop. Security policy owners gain a versioned, central CWE-to-severity policy. Security reviewers and CI consumers gain a bounded 0-100 AppSec project score with a two-condition gate.
**Impact.** Today `findings.PatternRule` carries a string `severity`, `quick_scan_findings` reads a module-level `PATTERN_RULES`, `calibration.py` only demotes ranking (floored at `MIN_CALIBRATED_PRIORITY = 0.1`, keyed on `tags[0]`), `benchmark.py:368` writes an inert `next_improvement_candidate="rules_or_language_hunter"`, and `harness.HarnessRuntime.run_stage` is a sync dict-state driver. This design turns rules into TOML data, removes `PatternRule.severity`, sources severity from a vendored verycode `CWE_Score.tsv`, computes a project score, makes the benchmark actionable, and confines HarnessX behind an optional, SHA-pinned, lazy-imported extra — while `dependencies = []` and `requires-python = ">=3.11"` stay intact and the **≥90% recall / <10% FP** detection gate holds at every phase.
### Goals
- Rules become versioned data (`ruleset/<lang>/*.toml`) with a loop-controlled `status` and no rule-local severity.
- One CWE = one governed severity, sourced from `policy/CWE_Score.tsv` via `resolve_severity()`.
- A single 0-100 `project_score` from `policy.severity × reachability`, with a two-condition CI gate.
- HarnessX adopted at the hunter loop, the verifier, and the meta-harness, behind `openultrasast[harnessx]`.
- A bounded self-improvement loop (`rule`, `policy` levers) that auto-reverts byte-for-byte under the gate.
- The governance + scoring + benchmark planes stay **zero-dependency** and **byte-identical by default** through Phases 0-2.
### Non-Goals
- The loop authoring or rewriting detection pattern text (human PR only).
- The loop editing the authoritative 0-5 CWE severity (verycode-upstream authority).
- A full single-step HarnessX rewrite of all deterministic stages and weight-level RL (deferred to optional Phase 5+).
- Changing the reachability vocabulary, which stays exactly `reachable | inferred-file-surface | unknown`.
- Dynamic/DAST scoring — report-only for a static tool.
## Boundary Commitments
### This Spec Owns
- The **RulesetStore** (`ruleset/store.py` + `ruleset/<lang>/*.toml`): `load_ruleset()`, the `PatternRule` schema (detection-only), and the `enabled|shadow|disabled` status vocabulary.
- The **PolicyStore** (`policy/verycode.py` + `policy/CWE_Score.tsv`): `load_policy()`, `CwePolicy`, `resolve_severity()`, and the CWE-120→CWE-121 remap.
- The **ScoreModel** (`scoring/project_score.py`): `SEV_WEIGHT`, `REACH_MULT`, `finding_penalty`, `project_score`, the score artifact, and the two-condition CI gate.
- The **loop ledger** (`.openultrasast/calibration/rule_policy.json`): the loop-owned overlay of `status`/evidence-floor/threshold/precision.
- The **HarnessX seam**: `harness/build_sast.py`, `HxScanOrchestrator`, the `MultiHookProcessor` SAST processors, the `SlotContractMixin`, and the `improve/` meta-harness driver.
- The **benchmark→signal** conversion: `rule_id` attribution, per-rule precision/recall, `rule_policy_delta`, and `rule_signals.json`.
### Out of Boundary
- The detection-pattern **content** of rules (authored by humans via PR; the loop may only toggle status / raise floors).
- The authoritative **0-5 severity** numbers in `CWE_Score.tsv` (verycode-upstream; PR gaps there).
- **Stdlib only** for the governance/scoring/benchmark planes: `tomllib`, `csv`, `json`, `math`, `dataclasses`.
- **HarnessX** only behind the `[harnessx]` extra, pinned to a commit SHA, imported lazily behind `_has_harnessx()`.
- The existing `HarnessRuntime` as the top-level sync driver through Phases 0-4 (not removed by this spec until Phase 5).
- The existing reachability vocabulary emitted by `findings.py` (consumed, never redefined).
### Revalidation Triggers
- **Contract shape changes** to `PatternRule` (removing `severity`), `CwePolicy`, `VerificationResult`, `BenchmarkMiss`/`BenchmarkFalsePositive` (adding `rule_id`), or the `project_score` artifact JSON.
- **Data ownership change**: severity moving from rule → policy; calibration re-keying from `tags[0]` → `rule_id`.
- **Dependency direction change**: any new import of HarnessX outside a lazy `_has_harnessx()` guard; any addition to core `dependencies`.
- **Runtime prerequisite change**: the startup CWE-resolution invariant (fail-loud if an enabled rule's CWE is unmapped); the `pip install --dry-run` audit gate before Phase 3.
- **Lever surface change**: any addition to `journal._VALID_LEVERS` or the `EvolveValidator` bounds.
## Architecture
### Existing Architecture Analysis
| Existing element | File | Constraint to respect / debt to fix |
|---|---|---|
| `HarnessRuntime.run_stage(name, callback)` | `harness.py:74` | Sync, dict `state` (`{"degradations": []}`), single `HarnessEvent` (`scan_id`), `ProcessorSpec.reads/writes` allow-list. Incompatible with HarnessX async-generator + message-window contract — **bridge, do not map**. Stays as the top-level driver through Phase 4. |
| `PatternRule.severity: str` + module-level `PATTERN_RULES` | `findings.py:31,34` | The fused defect: regex carries its own severity string ("high"/"medium"/"low") and is the only volume control. **Remove `severity`; externalize patterns to TOML.** |
| `calibrate_rankings` demotion | `calibration.py:146` | `max(MIN_CALIBRATED_PRIORITY, calibrated - 0.5)`, keyed on `finding.tags[0]` (`calibration.py:262`). Demotes ranking, never the rule. **Re-key to `rule_id`; write status flips to ledger.** |
| `verify_findings` rejects on `reachability_status == "unknown"` | `verification.py:91` | Structural verifier. Keep as **zero-dep fallback**; add LLM-judge path. |
| `benchmark.py:368``next_improvement_candidate="rules_or_language_hunter"` | `benchmark.py` | Inert stub written to `calibration_records.json`. **Make actionable** via `rule_id` attribution + `rule_policy_delta`. |
| `cli.py``run_stage(...)` call chain | `cli.py:104-145` | The 13-stage sync pipeline. Stages migrate to HarnessX hooks incrementally; the hunter pool first. |
### Architecture Pattern & Boundary Map
**Selected pattern.** Three planes with one-directional authority: **governance data drives execution; execution emits signals; the improvement loop edits governance under a gate.** Nothing in execution mutates governance directly. The governance + scoring planes are **pure stdlib, zero HarnessX** (non-negotiable, so CI gating runs in the default install). HarnessX lives only in the execution and improvement planes, behind the extra.
**Boundaries.** RulesetStore detects, PolicyStore decides severity/scope, ScoreModel optimizes — three owners, three change channels (human PR for patterns/severity; loop-via-`EvolveValidator` for status/floors/score-constants).
**New components rationale.** PolicyStore/ScoreModel/RulesetStore exist to break the fused regex defect; `HxScanOrchestrator`/`VerifyProcessor` exist to host the agent loop and judge; `improve/evolve.py`+`EvolveValidator` exist to make adaptation safe (bounded, gated, reversible). **Steering compliance:** zero-dep default install preserved; detection gate kept as a hard constraint separate from the score reward.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
-`calibration.py` — re-key from `tags[0]` (line 262) to `rule_id`; write `rule_policy.json` status flips, not only ranking demotion (line 146).
-`benchmark.py` — add `rule_id` to `BenchmarkMiss`/`BenchmarkFalsePositive`; per-rule precision/recall in `BenchmarkMetrics`; emit `rule_policy_delta`; replace the inert stub at line 368; add `BenchEvaluator` subclass.
-`verification.py` — add LLM-judge path (`VerifyProcessor` over `llm_judge` helpers); keep structural verifier (`reachability_status == "unknown"` reject, line 91) as zero-dep fallback.
The top-level driver stays sync (`HarnessRuntime`); only the `hunter_pool` stage opens an async surface via `asyncio.run`. Deterministic stages run as `skip_model=True` processors carrying work in typed slots; the `SlotContractMixin` validates `reads_slots`/`writes_slots` on every hook (preserving the `ProcessorSpec.reads/writes` discipline). If a HarnessX capability is unavailable at runtime, the stage falls back to its existing equivalent and records a degradation in the manifest.
J -->|no| K2["VerifyProcessor: llm_judge build/parse/call<br/>(structural fallback if extra absent)"]
K --> L["record_calibration: lesson/missing -> FalsePositiveLearning<br/>+ per-rule outcome -> rule_policy.json"]
K2 --> L
L --> M["on_task_end: ScoreProcessor -> project_score + gate"]
M --> N["report / sarif / manifest (incl. score block)"]
```
### Self-improvement loop
The benchmark is the **regression suite governing the data stores**. The objective is `maximize project_score subject to recall ≥ 90% AND FP < 10%` as a hard feasibility constraint — kept separate so the loop cannot "improve" the number by shadowing rules that hurt real security. Every change is `shadow`-staged first, novelty/evidence/replay-gated, and reverted byte-for-byte on any failure.
```mermaid
sequenceDiagram
participant B as BenchmarkHarness
participant O as improve/evolve.py
participant M as MetaAgent.evolve()
participant V as EvolveValidator (_rules/_policy)
participant J as journal (compute_attribution/fill_gating)
| 1.1, 1.4 | Hunter pool on real `Harness.run()`; async confined to one stage | `HxScanOrchestrator` | `run.py:HxScanOrchestrator.run()` | Scan pipeline |
-`status ∈ {enabled, shadow, disabled}` is the volume control; the loop overlays `status`/`min_evidence_level`/`precision_estimate` from `rule_policy.json`.
-`load_policy() -> dict[str, CwePolicy]` keyed `"CWE-NNN"` from `CWE_Score.tsv` (167 CWEs; severity column header is literally `"Flaw Severity "` with a trailing space — parse defensively).
-`resolve_severity(policy, rule.cwe, target, ranking)` replaces every literal `rule.severity` read. **Policy always wins**; any legacy rule severity string is discarded.
- **Startup invariant (fail-loud):** every `enabled` rule's `cwe` must resolve in PolicyStore, else the scan aborts. `dynamic`-only CWEs are report-only.
defassert_rules_resolve(rules,policy)->None:# raises at startup if an enabled rule's CWE is unmapped
...
```
- **Invariants:** one CWE ⇒ one severity; two rules for the same CWE always receive identical severity.
### Scoring Layer
#### ScoreModel
| Field | Detail |
|-------|--------|
| Intent | Compute a 0-100 project score and the two-condition gate verdict |
| Requirements | 5.1-5.9 |
**Responsibilities & Constraints**
- Pure stdlib (`math`), always on, zero-dep — runs in the default install with no HarnessX.
- Excludes `shadow`-status findings before computing; assigns 0 penalty to dynamic-only/unmapped CWEs (reports them instead).
- The `REACH_MULT` keys are exactly the three values `findings.py` emits; the reachability multiplier is the **FP-calibration knob** (a confirmed FP lowers effective reachability, never deletes a rule).
- Emits the scan-level artifact and the manifest score block; computes the hard gate (sev5 + reachable) and the score gate (advisory-first).
- **Postconditions:** artifact JSON (see Data Models) written next to per-finding output and merged into the manifest.
### Execution Layer
#### HxScanOrchestrator
| Field | Detail |
|-------|--------|
| Intent | Run the hunter pool on a real HarnessX agent loop, confining async to one stage |
| Requirements | 1.1, 1.4, 1.7, 1.8 |
**Responsibilities & Constraints**
- Invoked from the existing `hunter_pool``run_stage` callback (`cli.py:122`); wraps `asyncio.run(model_config.agentic(harness_config).run(ScanTask(...)))`.
-`build_sast()` mirrors `examples/coding/build_coding`; `ModelConfig(main=hunter, judge=…, ranker=…, patcher=…).agentic(cfg)` keeps model info out of `HarnessConfig`.
- Produces a `StatefulTrajectory` per target; rewards backfilled from verifier verdicts. On capability-missing, falls back to `run_hunter_pool()` and appends to manifest `degradations`.
- Hook mapping: `stage_start`→`on_step_start`, `stage_end`→`on_step_end`, `scan_id`→`session_id`/`run_id`; `before/after_model|tool` direct. `HARNESSX_CONTRACT_MODE` (message-window) + `OUS_SLOT_CONTRACT` (key-level via `SlotContractMixin`) split the old single `contract_mode`.
#### PolicyScoringProcessor
| Field | Detail |
|-------|--------|
| Intent | Resolve policy severity per finding and make the emission decision |
- Drops `shadow` findings from the reported set and the score input while recording their outcome for precision tracking.
- Applies `min_emit_priority`/`min_emit_precision` thresholds — the new volume controls that replace "every regex match becomes a finding".
**Contracts**: Event [x]
#### VerifyProcessor
| Field | Detail |
|-------|--------|
| Intent | LLM-judge verification with a zero-dep structural fallback |
| Requirements | 1.2 |
**Responsibilities & Constraints**
- Wraps `build_judge_prompt`/`parse_judge_response`/`_call_judge` (`llm_judge.py`) as a `judge`-role sub-harness; maps the strict-JSON verdict (`plausible|unsupported|hedging|…` + `lesson`/`missing`) to `VerificationResult`.
- **Fallback:** the existing structural verifier (`verification.py:91`, rejects on `reachability_status == "unknown"`) is used verbatim when the extra is absent or the judge is unavailable.
**Contracts**: Event [x]
### Improvement Layer
#### EvolveValidator (`_rules` / `_policy`)
| Field | Detail |
|-------|--------|
| Intent | Enforce the bounded-lever safety contract on every loop edit |
| Requirements | 3.2-3.8 |
**Responsibilities & Constraints** (the gate — runs in the *validity* phase beside `_contract`/`_replay`)
-`_rules`: edit parses; `status ∈ {enabled,shadow,disabled}`; `tighten`/`loosen` adjusts only threshold/evidence-floor **within configured bounds**; `add` doesn't duplicate a `rule_id`; **every `cwe` resolves in PolicyStore**; **pattern text unchanged** (reject otherwise); **no `enabled→disabled` jump** (require `shadow`).
-`_policy`: touches only evidence floor, scope, `K`/`MIN_SCORE` within bounds; **never the 0-5 severity**.
- On any failure: `raise StrictValidationError(kind="rule_change")` → the whole round is rejected.
**Contracts**: Service [x]
##### Service Interface
```python
_VALID_LEVERS=frozenset({
"configuration","control","action","instruction",
"rule",# status/evidence-floor/threshold in ruleset/ — NEVER pattern text
"policy",# evidence floor + scope + K/MIN_SCORE — NEVER the 0-5 CWE severity
| Intent | Drive benchmark→signal→evolve→re-benchmark→accept/revert under the gate |
| Requirements | 4.1-4.8, 6.3 |
**Responsibilities & Constraints**
- Converts `rule_policy_delta` + FP learnings → `rule_signals.json` (recall side `signal="miss"`; precision side `signal="verifier_reject"|"fp"`); calls `MetaAgent.evolve()`.
- **Acceptance gate (hard):** round accepted **only if** aggregate recall ≥ 90% **AND** FP < 10% **AND**`project_score` did not regress on the held set **AND** no unpredicted regression; otherwise `fill_gating(outcome="reverted")` restores config byte-for-byte.
- Precision-dragging rules are **auto-shadowed**, never deleted; coverage gaps nominate a rule via `rule_signals.json` (pattern authoring stays human-PR).
- Objective: `maximize project_score s.t. recall/FP gate` — score is the reward, the gate is the feasibility region; never folded together.
min_evidence_level="static_corroboration"# loop may raise within bounds
precision_estimate=0.62# backfilled from benchmarks
version="3"
```
Status semantics: `enabled` fires/scores/reports; `shadow` fires and records outcomes but is excluded from report and score (mandatory staging state for any loop-proposed change); `disabled` does not fire.
Loop ledger (`rule_policy.json`) — loop-owned overlay, re-keyed on `rule_id`:
- **Authority direction:** governance data is the source of truth; execution reads it, emits findings/signals; only the loop (via `EvolveValidator`) writes `status`/floors/score-constants back, never the pattern or the 0-5 severity.
- **Temporal/versioning:** `PatternRule.version` and `rule_policy.json.round` give per-rule audit; the journal records each lever edit; `fill_gating(reverted)` restores byte-for-byte.
- **Serialization (resume-safe):** all scan slots (`sast.targets`, `sast.findings`, `sast.rankings`, verifications) use plain dicts/dataclasses with `_target_` registration so they survive `wake()`.
## Error Handling
### Error Strategy
| Error type | Condition | Response |
|---|---|---|
| **Startup config error** | An `enabled` rule's `cwe` is unmapped in PolicyStore | `assert_rules_resolve` raises; **fail loud**, abort scan before any work (Req 2.7) |
| **Malformed ruleset/policy TOML** | `tomllib` parse failure | Fail loud at load; CI catches before merge |
| **Acceptance-gate failure** | recall<90% OR FP≥10% OR score regressed OR unpredicted regression | `fill_gating(reverted)` restores config byte-for-byte (Req 4.5) |
| **HarnessX capability missing at runtime** | Extra absent or sub-harness unavailable | Stage falls back to existing equivalent; record degradation in manifest (Req 1.8) |
| **Slot contract violation** | Stage reads/writes an undeclared slot | `SlotContractMixin` raises under `OUS_SLOT_CONTRACT=strict`; warns under `warn` (Req 1.5) |
| **Slot serialization loss on resume** | Non-`_target_` slot restores as `None` | Per-slot-type round-trip test guards; `_target_` registration required |
### Monitoring
-`EpisodeMetricsProcessor.episode_summary` → manifest metrics block; `OTelProcessor` OTLP export when configured.
- The score artifact `unmapped_cwe` list surfaces policy-completeness drift; the journal changeset table surfaces every rule/policy lever edit per round.
## Testing Strategy
### Unit Tests
-`load_policy()` parses the `"Flaw Severity "` trailing-space header and yields 167 `CwePolicy` entries with correct `static`/`dynamic` flags.
-`project_score()` boundary cases: P=0 → 100; one severity-5 reachable finding → ~43; one low finding → ~97; shadow findings excluded.
-`resolve_severity()` returns policy severity (not the legacy rule string) and is identical for two rules sharing a CWE.
-`finding_penalty()` returns 0.0 for dynamic-only and unmapped CWEs.
### Integration Tests
- **Byte-identical baseline (Req 6.2):** with the default ruleset and an empty `rule_policy.json`, `benchmark.py` output matches the pre-feature golden corpus exactly.
- **Startup invariant (Req 2.7):** an enabled rule with an unmapped CWE aborts the scan with a loud error.
- **Self-improvement round (Req 4.4/4.5):** a deliberately noisy rule is auto-shadowed when it drags precision; a round that regresses recall is reverted byte-for-byte.
- **Capability fallback (Req 1.8):** extra absent → `quick`/`standard` run through the sync driver unchanged; degradation recorded when a HX stage is requested but unavailable.
- **Slot round-trip (Req on resume):** each scan slot type serializes and restores without loss after `wake()`.
### E2E / Gate Tests
- **CI detection gate (Req 6.1):** the pipeline fails the merge if recall<90% or FP≥10% on the corpus.
- **Two-condition score gate (Req 5.7/5.8):** hard gate fails on a sev5+reachable finding; score gate is advisory until `K`/`MIN_SCORE` are calibrated, then blocking.
- **Zero-dep guard (Req 7.5):** with HarnessX absent, governance/scoring/benchmark planes operate and CI gating runs in the default install.
### Performance / Tolerance
- **HX tolerance (Req 6.4):** an enabled HarnessX-backed stage keeps recall/FP within the configured tolerance of the zero-dep baseline.
- Hunter-loop cost/token budgets enforced by `CostGuardProcessor`/`TokenBudgetProcessor`.
## Security Considerations
- **Supply-chain posture (Req 7):** core `dependencies = []` is preserved; HarnessX ships only as a SHA-pinned optional extra, lazy-imported behind `_has_harnessx()`. A `pip install --dry-run` audit runs before Phase 3; if the realized transitive graph pulls browser binaries / Docker SDK / a FastAPI web stack as *mandatory* deps and maintainers judge it unacceptable for a supply-chain auditing tool, the pre-specified **vendored lean subset** (Option C) is switched in under the same extra name — a packaging change, not a redesign. Submodule init is **non-recursive** so the large `verl` RL fork is never pulled.
- **Loop containment:** the meta-agent can never invent or rewrite detectors (no pattern edits) or rewrite the severity authority (no 0-5 edits); `shadow`-first staging, novelty/evidence/replay gates, and byte-for-byte revert bound its blast radius — directly preventing the "too influential" defect from re-entering via the loop.
- **Gate integrity:** the recall/FP gate is a hard constraint that is never folded into the score reward, so the loop cannot "improve" the number by shadowing rules that hurt real security.
## Migration Strategy
Every phase is independently shippable, test-gated, and reversible. Phases 0-2 deliver governance value at **zero dependency cost** and stay byte-identical by default. Phases 3-4 are the HarnessX adoption behind the extra. Phase 5 is the optional native end-state.
| **5 — Full harness + RL (optional)** | Migrate deterministic stages to single-step harness + `SlotContractMixin`; retire `HarnessRuntime`; promote score gate to blocking; optional `rl/`. | **opt-in** | golden-diff vs Phase 3 empty; gate held | per-stage revert |
**Rollback triggers.** Any phase that drops recall<90% or raises FP≥10% on the corpus blocks the merge; any self-improvement round failing the acceptance gate reverts byte-for-byte; an unacceptable `pip --dry-run` graph switches Phase 3 to the vendored subset.
