From 9b399fdc600f430f0c44dd7374a27edaa4cdd25e Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 22 Jun 2026 12:07:54 +0200 Subject: [PATCH] docs(spec): kick off harnessx-self-improving-rulesets Kiro spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Spec-driven plan to (1) adopt HarnessX (Darwin-Agent/HarnessX) directly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop — and migrate scan flows onto it stage-by-stage; (2) make detection rules versioned, centrally-governed data whose severity comes from a vendored verycode CWE policy (CWE_Score.tsv) instead of the rule itself, fixing "rules too influential / not centrally managed"; (3) adapt rules/policy inside HarnessX's MetaAgent.evolve loop under bounded levers with replay/novelty/ validator gates; (4) make benchmarks actionable per-rule signals with a hard accept gate and byte-for-byte revert; (5) compute a 0-100 verycode-style project score with a two-condition CI gate. Key decisions captured in research.md: HarnessX is an LLM-agent harness (not a generic stage pipeline), so it is adopted only on the agentic plane behind an optional SHA-pinned extra while the governance/scoring/benchmark planes stay zero-dependency; the >=90% recall / <10% FP detection gate is a hard constraint separate from the score reward and must hold at every phase; CWE-120 is unmapped in verycode so C/C++ buffer rules remap to CWE-121 (severity 5). Artifacts: spec.json (phase tasks-generated, approvals pending), requirements.md (7 requirements, EARS), design.md (three-plane architecture), tasks.md (4 phases), research.md (D1-D9 decisions). Awaiting human approval per the kiro 3-phase gate; not yet implemented. Co-Authored-By: Claude Opus 4.8 --- .../design.md | 611 ++++++++++++++++++ .../requirements.md | 115 ++++ .../research.md | 191 ++++++ .../spec.json | 22 + .../harnessx-self-improving-rulesets/tasks.md | 275 ++++++++ 5 files changed, 1214 insertions(+) create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/design.md create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/requirements.md create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/research.md create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/spec.json create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/tasks.md diff --git a/.kiro/specs/harnessx-self-improving-rulesets/design.md b/.kiro/specs/harnessx-self-improving-rulesets/design.md new file mode 100644 index 0000000..8084947 --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/design.md @@ -0,0 +1,611 @@ +# Design Document + +## Overview + +**Purpose.** This feature splits OpenUltraSAST's single overloaded regex layer — which today is detector, severity authority, and volume knob at once — into three governed owners: **rules detect**, a **central CWE policy decides severity and scope**, and a **single project score** becomes the optimization target. On top of that governance plane it adopts the HarnessX runtime exactly where it pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds. + +**Users.** OpenUltraSAST maintainers gain a staged, reversible HarnessX adoption and an automatic, gated rule/policy tuning loop. Security policy owners gain a versioned, central CWE-to-severity policy. Security reviewers and CI consumers gain a bounded 0-100 AppSec project score with a two-condition gate. + +**Impact.** Today `findings.PatternRule` carries a string `severity`, `quick_scan_findings` reads a module-level `PATTERN_RULES`, `calibration.py` only demotes ranking (floored at `MIN_CALIBRATED_PRIORITY = 0.1`, keyed on `tags[0]`), `benchmark.py:368` writes an inert `next_improvement_candidate="rules_or_language_hunter"`, and `harness.HarnessRuntime.run_stage` is a sync dict-state driver. This design turns rules into TOML data, removes `PatternRule.severity`, sources severity from a vendored verycode `CWE_Score.tsv`, computes a project score, makes the benchmark actionable, and confines HarnessX behind an optional, SHA-pinned, lazy-imported extra — while `dependencies = []` and `requires-python = ">=3.11"` stay intact and the **≥90% recall / <10% FP** detection gate holds at every phase. + +### Goals + +- Rules become versioned data (`ruleset//*.toml`) with a loop-controlled `status` and no rule-local severity. +- One CWE = one governed severity, sourced from `policy/CWE_Score.tsv` via `resolve_severity()`. +- A single 0-100 `project_score` from `policy.severity × reachability`, with a two-condition CI gate. +- HarnessX adopted at the hunter loop, the verifier, and the meta-harness, behind `openultrasast[harnessx]`. +- A bounded self-improvement loop (`rule`, `policy` levers) that auto-reverts byte-for-byte under the gate. +- The governance + scoring + benchmark planes stay **zero-dependency** and **byte-identical by default** through Phases 0-2. + +### Non-Goals + +- The loop authoring or rewriting detection pattern text (human PR only). +- The loop editing the authoritative 0-5 CWE severity (verycode-upstream authority). +- A full single-step HarnessX rewrite of all deterministic stages and weight-level RL (deferred to optional Phase 5+). +- Changing the reachability vocabulary, which stays exactly `reachable | inferred-file-surface | unknown`. +- Dynamic/DAST scoring — report-only for a static tool. + +## Boundary Commitments + +### This Spec Owns + +- The **RulesetStore** (`ruleset/store.py` + `ruleset//*.toml`): `load_ruleset()`, the `PatternRule` schema (detection-only), and the `enabled|shadow|disabled` status vocabulary. +- The **PolicyStore** (`policy/verycode.py` + `policy/CWE_Score.tsv`): `load_policy()`, `CwePolicy`, `resolve_severity()`, and the CWE-120→CWE-121 remap. +- The **ScoreModel** (`scoring/project_score.py`): `SEV_WEIGHT`, `REACH_MULT`, `finding_penalty`, `project_score`, the score artifact, and the two-condition CI gate. +- The **loop ledger** (`.openultrasast/calibration/rule_policy.json`): the loop-owned overlay of `status`/evidence-floor/threshold/precision. +- The **HarnessX seam**: `harness/build_sast.py`, `HxScanOrchestrator`, the `MultiHookProcessor` SAST processors, the `SlotContractMixin`, and the `improve/` meta-harness driver. +- The **benchmark→signal** conversion: `rule_id` attribution, per-rule precision/recall, `rule_policy_delta`, and `rule_signals.json`. + +### Out of Boundary + +- The detection-pattern **content** of rules (authored by humans via PR; the loop may only toggle status / raise floors). +- The authoritative **0-5 severity** numbers in `CWE_Score.tsv` (verycode-upstream; PR gaps there). +- Upstream **HarnessX internals** beyond the minimal meta-harness patch (`_VALID_LEVERS`, `_render_task_brief`, `EvolveValidator`, changeset buckets, `contract_autocheck`). +- The DAST/dynamic execution engine; dynamic-only CWEs are report-only here. +- Retiring `HarnessRuntime` and adopting `rl/` — explicitly Phase 5+, optional. + +### Allowed Dependencies + +- **Stdlib only** for the governance/scoring/benchmark planes: `tomllib`, `csv`, `json`, `math`, `dataclasses`. +- **HarnessX** only behind the `[harnessx]` extra, pinned to a commit SHA, imported lazily behind `_has_harnessx()`. +- The existing `HarnessRuntime` as the top-level sync driver through Phases 0-4 (not removed by this spec until Phase 5). +- The existing reachability vocabulary emitted by `findings.py` (consumed, never redefined). + +### Revalidation Triggers + +- **Contract shape changes** to `PatternRule` (removing `severity`), `CwePolicy`, `VerificationResult`, `BenchmarkMiss`/`BenchmarkFalsePositive` (adding `rule_id`), or the `project_score` artifact JSON. +- **Data ownership change**: severity moving from rule → policy; calibration re-keying from `tags[0]` → `rule_id`. +- **Dependency direction change**: any new import of HarnessX outside a lazy `_has_harnessx()` guard; any addition to core `dependencies`. +- **Runtime prerequisite change**: the startup CWE-resolution invariant (fail-loud if an enabled rule's CWE is unmapped); the `pip install --dry-run` audit gate before Phase 3. +- **Lever surface change**: any addition to `journal._VALID_LEVERS` or the `EvolveValidator` bounds. + +## Architecture + +### Existing Architecture Analysis + +| Existing element | File | Constraint to respect / debt to fix | +|---|---|---| +| `HarnessRuntime.run_stage(name, callback)` | `harness.py:74` | Sync, dict `state` (`{"degradations": []}`), single `HarnessEvent` (`scan_id`), `ProcessorSpec.reads/writes` allow-list. Incompatible with HarnessX async-generator + message-window contract — **bridge, do not map**. Stays as the top-level driver through Phase 4. | +| `PatternRule.severity: str` + module-level `PATTERN_RULES` | `findings.py:31,34` | The fused defect: regex carries its own severity string ("high"/"medium"/"low") and is the only volume control. **Remove `severity`; externalize patterns to TOML.** | +| `quick_scan_findings(root, targets, rankings)` | `findings.py` / `cli.py:126` | Reads `PATTERN_RULES` implicitly. Becomes `quick_scan_findings(..., ruleset)` wired like `static_hints`. | +| `calibrate_rankings` demotion | `calibration.py:146` | `max(MIN_CALIBRATED_PRIORITY, calibrated - 0.5)`, keyed on `finding.tags[0]` (`calibration.py:262`). Demotes ranking, never the rule. **Re-key to `rule_id`; write status flips to ledger.** | +| `verify_findings` rejects on `reachability_status == "unknown"` | `verification.py:91` | Structural verifier. Keep as **zero-dep fallback**; add LLM-judge path. | +| `benchmark.py:368` `next_improvement_candidate="rules_or_language_hunter"` | `benchmark.py` | Inert stub written to `calibration_records.json`. **Make actionable** via `rule_id` attribution + `rule_policy_delta`. | +| `cli.py` `run_stage(...)` call chain | `cli.py:104-145` | The 13-stage sync pipeline. Stages migrate to HarnessX hooks incrementally; the hunter pool first. | + +### Architecture Pattern & Boundary Map + +**Selected pattern.** Three planes with one-directional authority: **governance data drives execution; execution emits signals; the improvement loop edits governance under a gate.** Nothing in execution mutates governance directly. The governance + scoring planes are **pure stdlib, zero HarnessX** (non-negotiable, so CI gating runs in the default install). HarnessX lives only in the execution and improvement planes, behind the extra. + +**Boundaries.** RulesetStore detects, PolicyStore decides severity/scope, ScoreModel optimizes — three owners, three change channels (human PR for patterns/severity; loop-via-`EvolveValidator` for status/floors/score-constants). + +```mermaid +flowchart TB + subgraph GOV["GOVERNANCE PLANE — stdlib, versioned, audited, ZERO-DEP"] + direction LR + POL["PolicyStore
policy/verycode.py
CWE_Score.tsv -> CwePolicy
severity 0-5, static/dynamic"] + RUL["RulesetStore
ruleset/store.py
*.toml: pattern, cwe, status"] + LED["rule_policy.json
loop-owned: status/floor/precision"] + SCO["ScoreModel
scoring/project_score.py
policy-weighted -> 0-100 + gate"] + FPL["FP ledger
false_positive_learnings.json"] + end + subgraph EXE["EXECUTION PLANE — HarnessX runtime, OPTIONAL EXTRA"] + direction LR + DET["Deterministic stages
skip_model MultiHookProcessors"] + HUNT["hunter_pool
HxScanOrchestrator (real LLM loop)"] + VER["VerifyProcessor
llm_judge helpers (+ structural fallback)"] + PSP["PolicyScoringProcessor + ScoreProcessor"] + end + subgraph IMP["IMPROVEMENT PLANE — meta_harness, OPTIONAL EXTRA"] + EVO["improve/evolve.py -> MetaAgent.evolve()
levers: rule | policy"] + BEN["BenchmarkHarness
recall/FP + per-rule attribution"] + end + POL --> DET & PSP & SCO + RUL --> DET & HUNT + LED --> RUL + DET --> HUNT --> VER --> PSP --> SCO + VER --> FPL + EXE -->|trajectories + verdicts| BEN + BEN -->|rule_signals.json + deltas| EVO + EVO -->|gated edits| RUL & POL & LED + SCO -->|gate verdict| CI[("CI gate")] + BEN -->|>=90% recall / <10% FP| CI +``` + +**New components rationale.** PolicyStore/ScoreModel/RulesetStore exist to break the fused regex defect; `HxScanOrchestrator`/`VerifyProcessor` exist to host the agent loop and judge; `improve/evolve.py`+`EvolveValidator` exist to make adaptation safe (bounded, gated, reversible). **Steering compliance:** zero-dep default install preserved; detection gate kept as a hard constraint separate from the score reward. + +### Technology Stack + +| Layer | Choice / Version | Role in Feature | Notes | +|-------|------------------|-----------------|-------| +| CLI | `cli.py` (`_run_scan`) | Wires ruleset/policy; Phase 3 routes hunter pool to `HxScanOrchestrator` | Unchanged entrypoints when extra absent | +| Governance | Python 3.11 stdlib (`tomllib`, `csv`, `dataclasses`) | RulesetStore + PolicyStore load | Zero-dep, fail-loud on unmapped CWE | +| Scoring | Python stdlib `math` | `project_score` exponential decay | Always on, zero-dep | +| Execution runtime | HarnessX `@ git+...@` (optional extra) | Hunter loop, judge verifier, processors | Lazy import behind `_has_harnessx()` | +| Self-improvement | HarnessX `meta_harness` (extra/vendored) | `MetaAgent.evolve` + `EvolveValidator` + journal | 5/6-hook minimal patch | +| Data / Storage | TOML rulesets, TSV policy, JSON ledger/artifacts | Versioned governance + loop overlay | All `_target_`/JSON-serializable for resume | +| Events | HarnessX lifecycle hooks (`on_task_start`…`on_task_end`) | Stage→hook mapping | `HarnessRuntime` sync driver retained Phases 0-4 | +| Packaging | `pyproject.toml` `[project.optional-dependencies]` | `harnessx` extra, pinned SHA | Core `dependencies = []` unchanged | + +## File Structure Plan + +### Directory Structure + +``` +src/openultrasast/ +├── ruleset/ +│ ├── store.py # load_ruleset()/write_ruleset(); PatternRule (NO severity); status/min_evidence_level/precision_estimate/version +│ ├── python/*.toml # rules-as-data, one [[rule]] per detector +│ ├── c/*.toml # incl. CWE-120 -> CWE-121 remapped buffer rules +│ └── /*.toml # same pattern per language +├── policy/ +│ ├── verycode.py # load_policy() -> dict[str, CwePolicy]; resolve_severity(); CwePolicy dataclass +│ └── CWE_Score.tsv # vendored verycode data (167 CWEs; header "Flaw Severity " has trailing space) +├── scoring/ +│ └── project_score.py # SEV_WEIGHT, REACH_MULT, finding_penalty(), project_score(), gate() +├── harness/ # Phase 3 (extra) +│ ├── build_sast.py # build_sast() factory mirrors examples/coding/build_coding +│ ├── task.py # ScanTask(BaseTask) +│ ├── run.py # HxScanOrchestrator: asyncio.run around model_config.agentic(cfg).run(...) +│ ├── slot_contract.py # SlotContractMixin: reads_slots/writes_slots on MultiHookProcessor +│ └── processors.py # SarifIngest/Preprocess/EntryPoint/Rank/Calibrate/QuickFindings/PolicyScoring/Verify/Score processors +└── improve/ # Phase 4 (extra) + ├── evolve.py # orchestrator: benchmark -> delta -> rule_signals.json -> MetaAgent.evolve() -> re-benchmark -> accept/revert + ├── rule_validator.py # EvolveValidator._rules / ._policy bound checks + └── rule_autocheck.py # ContractAutoCheckProcessor _check_rule/_check_policy branches + +.openultrasast/calibration/ +└── rule_policy.json # loop-owned ledger: status/threshold/evidence-floor/precision per rule_id +``` + +### Modified Files + +- `findings.py` — extract `PATTERN_RULES` to TOML; **delete `PatternRule.severity`**; `quick_scan_findings(root, targets, rankings, ruleset)` takes ruleset arg. +- `calibration.py` — re-key from `tags[0]` (line 262) to `rule_id`; write `rule_policy.json` status flips, not only ranking demotion (line 146). +- `benchmark.py` — add `rule_id` to `BenchmarkMiss`/`BenchmarkFalsePositive`; per-rule precision/recall in `BenchmarkMetrics`; emit `rule_policy_delta`; replace the inert stub at line 368; add `BenchEvaluator` subclass. +- `verification.py` — add LLM-judge path (`VerifyProcessor` over `llm_judge` helpers); keep structural verifier (`reachability_status == "unknown"` reject, line 91) as zero-dep fallback. +- `config.py` — ruleset/policy/score paths + gate thresholds (`K`, `MIN_SCORE`, `min_emit_priority`, `min_emit_precision`). +- `cli.py` — `_run_scan` wires ruleset/policy; Phase 3 routes the `hunter_pool` `run_stage` call (line 122) to `HxScanOrchestrator`. +- `harness.py` — retired in Phase 5 only. +- `pyproject.toml` — add `[project.optional-dependencies] harnessx = ["harnessx @ git+https://github.com/Darwin-Agent/HarnessX.git@"]`; core `dependencies = []` unchanged. +- `harnessx/meta_harness/{journal.py, agent.py, validate_workflow.py, processors/contract_autocheck.py}` — vendored/patched seam (Phase 4). + +## System Flows + +### Scan pipeline on HarnessX (Phase 3 seam) + +The top-level driver stays sync (`HarnessRuntime`); only the `hunter_pool` stage opens an async surface via `asyncio.run`. Deterministic stages run as `skip_model=True` processors carrying work in typed slots; the `SlotContractMixin` validates `reads_slots`/`writes_slots` on every hook (preserving the `ProcessorSpec.reads/writes` discipline). If a HarnessX capability is unavailable at runtime, the stage falls back to its existing equivalent and records a degradation in the manifest. + +```mermaid +flowchart TD + A["on_task_start: SarifIngestProcessor
-> slot sast.static_hints"] --> B["Preprocess -> sast.targets"] + B --> C["EntryPoint -> sast.entry_points"] + C --> D["on_step_start: Rank -> sast.rankings"] + D --> E["Calibrate (MemoryRetrieval pattern over FP ledger)
demotes sast.rankings"] + E --> F{"HarnessX extra
present & capable?"} + F -->|yes| G["hunter_pool: HxScanOrchestrator
asyncio.run(model_config.agentic(cfg).run(ScanTask))
control/tools/obs processors -> StatefulTrajectory"] + F -->|no / capability missing| G2["fallback run_hunter_pool()
record degradation in manifest"] + G --> H["QuickFindings: regex over load_ruleset()
-> sast.findings (NO severity yet)"] + G2 --> H + H --> I["PolicyScoringProcessor: resolve_severity(policy, rule.cwe)
+ emission decision; drop shadow findings"] + I --> J{"reachability_status
== unknown?"} + J -->|yes| K["VerifyProcessor: REJECTED (structural fallback)
or llm_judge verdict"] + J -->|no| K2["VerifyProcessor: llm_judge build/parse/call
(structural fallback if extra absent)"] + K --> L["record_calibration: lesson/missing -> FalsePositiveLearning
+ per-rule outcome -> rule_policy.json"] + K2 --> L + L --> M["on_task_end: ScoreProcessor -> project_score + gate"] + M --> N["report / sarif / manifest (incl. score block)"] +``` + +### Self-improvement loop + +The benchmark is the **regression suite governing the data stores**. The objective is `maximize project_score subject to recall ≥ 90% AND FP < 10%` as a hard feasibility constraint — kept separate so the loop cannot "improve" the number by shadowing rules that hurt real security. Every change is `shadow`-staged first, novelty/evidence/replay-gated, and reverted byte-for-byte on any failure. + +```mermaid +sequenceDiagram + participant B as BenchmarkHarness + participant O as improve/evolve.py + participant M as MetaAgent.evolve() + participant V as EvolveValidator (_rules/_policy) + participant J as journal (compute_attribution/fill_gating) + participant L as rule_policy.json + B->>O: misses(recall) + FPs/verifier-rejects(precision) + per-rule metrics + O->>O: write rule_signals.json [{rule_id, signal, cwe, evidence, suggested_action}] + O->>M: evolve(brief + rule_signals.json) + M->>M: write ruleset/*.toml + policy/overrides.toml + candidates.md + journal Round N + M->>V: validity(canon -> contract -> _rules/_policy -> replay) + policy(novelty + evidence) + alt validation fails + V-->>M: StrictValidationError(kind="rule_change") -> round rejected + else passes + V->>B: re-run benchmark on candidate + B->>J: passed_now vs passed_before + J->>J: compute_attribution(predicted): flipped/still_F/regressed/still_T/absent + alt recall<90% OR FP>=10% OR score regressed OR unpredicted regression + J->>L: fill_gating(reverted) -> restore config byte-for-byte + else + J->>L: fill_gating(accepted) -> persist status/floor/precision + end + end +``` + +## Requirements Traceability + +| Requirement | Summary | Components | Interfaces | Flows | +|-------------|---------|------------|------------|-------| +| 1.1, 1.4 | Hunter pool on real `Harness.run()`; async confined to one stage | `HxScanOrchestrator` | `run.py:HxScanOrchestrator.run()` | Scan pipeline | +| 1.2 | Verify via llm-judge + structural fallback | `VerifyProcessor` | `processors.py:VerifyProcessor` | Scan pipeline | +| 1.3 | Extra absent → unchanged quick/standard | `cli._run_scan`, `_has_harnessx()` | capability guard | Scan pipeline (`F` branch) | +| 1.5 | Slot allow-list validated per hook | `SlotContractMixin` | `slot_contract.py` | Scan pipeline | +| 1.6 | Deterministic stages `skip_model` | SAST processors | `MultiHookProcessor(skip_model=True)` | Scan pipeline | +| 1.7 | Event/`scan_id` rename to HarnessX hooks/`run_id` | `build_sast`, journal | hook map table | Scan pipeline | +| 1.8 | Capability-missing fallback + manifest degradation | `HxScanOrchestrator` | manifest `degradations` | Scan pipeline (`G2`) | +| 2.1, 2.4 | Rules-as-data; 3 statuses; no rule severity | RulesetStore | `load_ruleset()`, `PatternRule` | — | +| 2.2, 2.3 | Severity from policy; legacy strings discarded | PolicyStore, `PolicyScoringProcessor` | `resolve_severity()` | Scan pipeline (`I`) | +| 2.5 | Shadow excluded from report+score | `PolicyScoringProcessor`, ScoreModel | emission filter | Scan pipeline (`I`,`M`) | +| 2.6 | CwePolicy 0-5 + static/dynamic; dynamic report-only | PolicyStore | `CwePolicy` | — | +| 2.7 | Fail-loud on unmapped enabled-rule CWE | PolicyStore startup invariant | `load_policy()` check | — | +| 2.8 | CWE-120 → CWE-121 remap | RulesetStore | TOML data | — | +| 2.9, 5.9, 7.5 | Governance/score/benchmark zero-dep | all GOV components | stdlib-only | — | +| 3.1 | Two levers `rule`/`policy` | `improve/evolve.py` | `_VALID_LEVERS` | Self-improvement | +| 3.2, 3.3 | No pattern text / no 0-5 severity edits | `EvolveValidator` | `_rules`/`_policy` | Self-improvement | +| 3.4 | No enabled→disabled jump (shadow mandatory) | `EvolveValidator._rules` | bound check | Self-improvement | +| 3.5 | Parse/status/bounds/dup/CWE-resolve checks | `EvolveValidator._rules` | validity phase | Self-improvement | +| 3.6 | Novelty gate on reverted re-proposals | `journal.check_novelty` | novelty gate | Self-improvement | +| 3.7 | Replay smoke gate (ruleset boots) | `run_synthetic_task_smoke_gate` | replay phase | Self-improvement | +| 3.8, 3.9 | `StrictValidationError`; journal each edit | `EvolveValidator`, journal | `fill_gating` | Self-improvement | +| 4.1, 4.2 | `rule_id` attribution; per-rule metrics; delta artifact | BenchmarkHarness | `rule_policy_delta` | Self-improvement | +| 4.3 | Delta → `rule_signals.json` (miss/fp) | `improve/evolve.py` | signal writer | Self-improvement | +| 4.4, 4.5 | Hard accept gate; byte-for-byte revert | `improve/evolve.py`, journal | `fill_gating(reverted)` | Self-improvement | +| 4.6 | Auto-shadow precision-draggers | `improve/evolve.py` | status flip | Self-improvement | +| 4.7 | Coverage-gap nominates rule (human PR) | `improve/evolve.py` | `rule_signals.json` | Self-improvement | +| 4.8, 6.5 | Score=reward, gate=feasibility (separate) | `improve/evolve.py` | objective split | Self-improvement | +| 5.1, 5.2 | 0-100 score; reachability multipliers | ScoreModel | `project_score()` | Scan pipeline (`M`) | +| 5.3 | Dynamic-only/unmapped → 0 penalty, report-only | ScoreModel | `finding_penalty()` | Scan pipeline (`M`) | +| 5.4 | Shadow excluded from score input | ScoreModel | pre-call filter | Scan pipeline (`M`) | +| 5.5 | Score artifact + manifest block | ScoreModel | score JSON | Scan pipeline (`N`) | +| 5.6 | FP lowers effective reachability, not delete | `calibration.py` | reachability knob | Self-improvement | +| 5.7 | Hard gate: sev5 + reachable → fail | CI Score Gate | `gate()` | — | +| 5.8 | Score gate advisory-first, then blocking | CI Score Gate | `gate()` | — | +| 6.1, 6.4, 6.6 | Per-merge recall/FP gate + HX tolerance | CI Pipeline | benchmark gate | — | +| 6.2 | Byte-identical baseline Phases 0-2 | BenchmarkHarness | golden diff | Migration | +| 6.3 | Gate must pass on re-benchmarked candidate | `improve/evolve.py` | accept gate | Self-improvement | +| 7.1, 7.6 | Empty core deps; pinned extra; non-recursive submodule | Packaging Config | `pyproject.toml` | Migration | +| 7.2 | Lazy import behind capability check | `_has_harnessx()` | guard | — | +| 7.3, 7.4, 7.7 | Dry-run audit; vendored fallback trigger; pre-specified | Packaging Config | audit gate | Migration (Phase 3 gate) | + +## Components and Interfaces + +| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts | +|-----------|--------------|--------|--------------|--------------------------|-----------| +| RulesetStore | Governance | Rules-as-data + status | 2.1,2.4,2.8 | tomllib (P0), rule_policy.json (P1) | State | +| PolicyStore | Governance | CWE→severity/scope authority | 2.2,2.6,2.7 | CWE_Score.tsv (P0) | State | +| ScoreModel | Scoring | 0-100 score + gate | 5.1-5.9 | PolicyStore (P0) | Batch, State | +| HxScanOrchestrator | Execution | Hunter loop on HarnessX | 1.1,1.4,1.8 | HarnessX (P0), RulesetStore (P1) | Service, Event | +| PolicyScoringProcessor | Execution | Resolve severity + emission | 2.2,2.3,2.5 | PolicyStore (P0) | Event | +| VerifyProcessor | Execution | LLM-judge + structural fallback | 1.2 | llm_judge (P1) | Event | +| EvolveValidator (`_rules`/`_policy`) | Improvement | Bound-check loop edits | 3.2-3.8 | journal (P0), PolicyStore (P1) | Service | +| improve/evolve.py | Improvement | Benchmark→signal→evolve→gate | 4.1-4.8,6.3 | BenchmarkHarness (P0), MetaAgent (P0) | Batch | + +### Governance Layer + +#### RulesetStore + +| Field | Detail | +|-------|--------| +| Intent | Load detection rules from versioned TOML; carry loop-controlled status; carry no severity | +| Requirements | 2.1, 2.4, 2.8, 2.9 | + +**Responsibilities & Constraints** +- `load_ruleset(path) -> tuple[PatternRule, ...]` replaces module-level `PATTERN_RULES`; loaded with `tomllib` (zero-dep). +- `PatternRule` carries `rule_id`, `languages`, `cwe`, `tags`, `pattern`, `status`, `min_evidence_level`, `precision_estimate`, `version` — **no `severity` field**. +- `status ∈ {enabled, shadow, disabled}` is the volume control; the loop overlays `status`/`min_evidence_level`/`precision_estimate` from `rule_policy.json`. +- C/C++ buffer rules (`c-unsafe-copy`, `c-unsafe-memory-copy`, `c-unsafe-scanf`) declare `cwe = "CWE-121"` (remapped from unmapped CWE-120). + +**Dependencies** — Inbound: `quick_scan_findings` (P0). Outbound: `rule_policy.json` overlay (P1). External: `tomllib` (P0). + +**Contracts**: State [x] + +##### Service Interface +```python +def load_ruleset(path: Path, ledger: Path | None = None) -> tuple[PatternRule, ...]: ... +def write_ruleset(path: Path, rules: Iterable[PatternRule]) -> None: ... + +@dataclass(frozen=True) +class PatternRule: + rule_id: str + languages: tuple[str, ...] + cwe: str # the ONLY policy join key + tags: tuple[str, ...] + pattern: str # human-PR-only; loop must never edit + status: str = "enabled" # enabled | shadow | disabled (loop-controlled) + min_evidence_level: str = "static_corroboration" + precision_estimate: float = 0.0 + version: str = "1" +``` +- **Preconditions:** every loaded rule has a non-empty `pattern` and a `CWE-NNN` `cwe`. +- **Postconditions:** ledger overlay applied; `status`/floors reflect `rule_policy.json`. +- **Invariants:** no `severity` attribute exists; `status` is one of the three literals. + +#### PolicyStore + +| Field | Detail | +|-------|--------| +| Intent | Authoritative CWE→severity/scope; resolve finding severity | +| Requirements | 2.2, 2.3, 2.6, 2.7, 2.8 | + +**Responsibilities & Constraints** +- `load_policy() -> dict[str, CwePolicy]` keyed `"CWE-NNN"` from `CWE_Score.tsv` (167 CWEs; severity column header is literally `"Flaw Severity "` with a trailing space — parse defensively). +- `resolve_severity(policy, rule.cwe, target, ranking)` replaces every literal `rule.severity` read. **Policy always wins**; any legacy rule severity string is discarded. +- **Startup invariant (fail-loud):** every `enabled` rule's `cwe` must resolve in PolicyStore, else the scan aborts. `dynamic`-only CWEs are report-only. + +**Dependencies** — Inbound: `PolicyScoringProcessor`, ScoreModel (P0). External: `csv`, `dataclasses` (P0). + +**Contracts**: State [x] + +##### Service Interface +```python +@dataclass(frozen=True) +class CwePolicy: + flaw_category: str + severity: int # 0..5 — AUTHORITATIVE; overrides any rule string; loop must never edit + static: bool # SAST scope gate + dynamic: bool # DAST scope — report out-of-scope, never scored by static tool + +def load_policy(tsv: Path = CWE_SCORE_TSV) -> dict[str, CwePolicy]: ... +def resolve_severity(policy: Mapping[str, CwePolicy], cwe: str, target, ranking) -> int: ... +def assert_rules_resolve(rules, policy) -> None: # raises at startup if an enabled rule's CWE is unmapped + ... +``` +- **Invariants:** one CWE ⇒ one severity; two rules for the same CWE always receive identical severity. + +### Scoring Layer + +#### ScoreModel + +| Field | Detail | +|-------|--------| +| Intent | Compute a 0-100 project score and the two-condition gate verdict | +| Requirements | 5.1-5.9 | + +**Responsibilities & Constraints** +- Pure stdlib (`math`), always on, zero-dep — runs in the default install with no HarnessX. +- Excludes `shadow`-status findings before computing; assigns 0 penalty to dynamic-only/unmapped CWEs (reports them instead). +- The `REACH_MULT` keys are exactly the three values `findings.py` emits; the reachability multiplier is the **FP-calibration knob** (a confirmed FP lowers effective reachability, never deletes a rule). +- Emits the scan-level artifact and the manifest score block; computes the hard gate (sev5 + reachable) and the score gate (advisory-first). + +**Contracts**: Batch [x] / State [x] + +##### Service Interface +```python +SEV_WEIGHT = {5: 50, 4: 25, 3: 10, 2: 2, 1: 1, 0: 0} +REACH_MULT = {"reachable": 1.0, "inferred-file-surface": 0.6, "unknown": 0.4} + +def finding_penalty(f, rule_cwe: str, policy) -> float: + pol = policy.get(rule_cwe) + if pol is None or not pol.static: # dynamic-only / unmapped -> report, don't score + return 0.0 + return SEV_WEIGHT[pol.severity] * REACH_MULT.get(f.reachability_status, 0.4) + +def project_score(findings, rule_cwe_by_id, policy, K: float = 60.0) -> int: + # shadow-status findings excluded before this call + P = sum(finding_penalty(f, rule_cwe_by_id[f.finding_id], policy) for f in findings) + return round(100 * math.exp(-P / K)) # P=0 -> 100; one critical-reachable -> ~43 + +def gate(findings, score, *, min_score: int, block_severity_reachable: int = 5, + blocking: bool) -> dict: ... +``` +- **Postconditions:** artifact JSON (see Data Models) written next to per-finding output and merged into the manifest. + +### Execution Layer + +#### HxScanOrchestrator + +| Field | Detail | +|-------|--------| +| Intent | Run the hunter pool on a real HarnessX agent loop, confining async to one stage | +| Requirements | 1.1, 1.4, 1.7, 1.8 | + +**Responsibilities & Constraints** +- Invoked from the existing `hunter_pool` `run_stage` callback (`cli.py:122`); wraps `asyncio.run(model_config.agentic(harness_config).run(ScanTask(...)))`. +- `build_sast()` mirrors `examples/coding/build_coding`; `ModelConfig(main=hunter, judge=…, ranker=…, patcher=…).agentic(cfg)` keeps model info out of `HarnessConfig`. +- Each hunter sub-harness registers `LoopDetectionProcessor`, `CostGuardProcessor`/`TokenBudgetProcessor`, `ToolFailureGuard`, `ToolFilterProcessor`, `SelfVerifyProcessor`, `EpisodeMetricsProcessor`/`OTelProcessor`/`CheckpointProcessor`. +- Produces a `StatefulTrajectory` per target; rewards backfilled from verifier verdicts. On capability-missing, falls back to `run_hunter_pool()` and appends to manifest `degradations`. + +**Dependencies** — External: HarnessX core + processors (P0, lazy). Inbound: `cli._run_scan` (P0). + +**Contracts**: Service [x] / Event [x] + +##### Event Contract +- Hook mapping: `stage_start`→`on_step_start`, `stage_end`→`on_step_end`, `scan_id`→`session_id`/`run_id`; `before/after_model|tool` direct. `HARNESSX_CONTRACT_MODE` (message-window) + `OUS_SLOT_CONTRACT` (key-level via `SlotContractMixin`) split the old single `contract_mode`. + +#### PolicyScoringProcessor + +| Field | Detail | +|-------|--------| +| Intent | Resolve policy severity per finding and make the emission decision | +| Requirements | 2.2, 2.3, 2.5 | + +**Responsibilities & Constraints** +- Runs `on_after_tool`/`on_step_end`; reads `sast.rankings` + PolicyStore; writes resolved severity + emit-decision. +- Drops `shadow` findings from the reported set and the score input while recording their outcome for precision tracking. +- Applies `min_emit_priority`/`min_emit_precision` thresholds — the new volume controls that replace "every regex match becomes a finding". + +**Contracts**: Event [x] + +#### VerifyProcessor + +| Field | Detail | +|-------|--------| +| Intent | LLM-judge verification with a zero-dep structural fallback | +| Requirements | 1.2 | + +**Responsibilities & Constraints** +- Wraps `build_judge_prompt`/`parse_judge_response`/`_call_judge` (`llm_judge.py`) as a `judge`-role sub-harness; maps the strict-JSON verdict (`plausible|unsupported|hedging|…` + `lesson`/`missing`) to `VerificationResult`. +- **Fallback:** the existing structural verifier (`verification.py:91`, rejects on `reachability_status == "unknown"`) is used verbatim when the extra is absent or the judge is unavailable. + +**Contracts**: Event [x] + +### Improvement Layer + +#### EvolveValidator (`_rules` / `_policy`) + +| Field | Detail | +|-------|--------| +| Intent | Enforce the bounded-lever safety contract on every loop edit | +| Requirements | 3.2-3.8 | + +**Responsibilities & Constraints** (the gate — runs in the *validity* phase beside `_contract`/`_replay`) +- `_rules`: edit parses; `status ∈ {enabled,shadow,disabled}`; `tighten`/`loosen` adjusts only threshold/evidence-floor **within configured bounds**; `add` doesn't duplicate a `rule_id`; **every `cwe` resolves in PolicyStore**; **pattern text unchanged** (reject otherwise); **no `enabled→disabled` jump** (require `shadow`). +- `_policy`: touches only evidence floor, scope, `K`/`MIN_SCORE` within bounds; **never the 0-5 severity**. +- On any failure: `raise StrictValidationError(kind="rule_change")` → the whole round is rejected. + +**Contracts**: Service [x] + +##### Service Interface +```python +_VALID_LEVERS = frozenset({ + "configuration", "control", "action", "instruction", + "rule", # status/evidence-floor/threshold in ruleset/ — NEVER pattern text + "policy", # evidence floor + scope + K/MIN_SCORE — NEVER the 0-5 CWE severity +}) + +class EvolveValidator: + def _rules(self, edit) -> None: ... # raises StrictValidationError(kind="rule_change") + def _policy(self, edit) -> None: ... +``` + +#### improve/evolve.py (orchestrator) + +| Field | Detail | +|-------|--------| +| Intent | Drive benchmark→signal→evolve→re-benchmark→accept/revert under the gate | +| Requirements | 4.1-4.8, 6.3 | + +**Responsibilities & Constraints** +- Converts `rule_policy_delta` + FP learnings → `rule_signals.json` (recall side `signal="miss"`; precision side `signal="verifier_reject"|"fp"`); calls `MetaAgent.evolve()`. +- **Acceptance gate (hard):** round accepted **only if** aggregate recall ≥ 90% **AND** FP < 10% **AND** `project_score` did not regress on the held set **AND** no unpredicted regression; otherwise `fill_gating(outcome="reverted")` restores config byte-for-byte. +- Precision-dragging rules are **auto-shadowed**, never deleted; coverage gaps nominate a rule via `rule_signals.json` (pattern authoring stays human-PR). +- Objective: `maximize project_score s.t. recall/FP gate` — score is the reward, the gate is the feasibility region; never folded together. + +**Contracts**: Batch [x] + +## Data Models + +### Ruleset policy schema (detection only — TOML) + +```toml +[[rule]] +rule_id = "python-sql-injection" +languages = ["python"] +cwe = "CWE-89" # the ONLY policy join key +tags = ["injection"] +pattern = '(\.execute)\s*\(\s*[^,)]*%' # human-PR-only +status = "enabled" # enabled | shadow | disabled (loop-controlled) +min_evidence_level = "static_corroboration" # loop may raise within bounds +precision_estimate = 0.62 # backfilled from benchmarks +version = "3" +``` + +Status semantics: `enabled` fires/scores/reports; `shadow` fires and records outcomes but is excluded from report and score (mandatory staging state for any loop-proposed change); `disabled` does not fire. + +Loop ledger (`rule_policy.json`) — loop-owned overlay, re-keyed on `rule_id`: +```json +{ "python-sql-injection": { "status": "shadow", "min_evidence_level": "dynamic_corroboration", + "precision_estimate": 0.41, "outcome": "accepted", "round": 7 } } +``` + +### Project-score schema (artifact + manifest block) + +```json +{ "project_score": 73, "max_severity": 5, "penalty_total": 30.0, + "by_category": {"Command or Argument Injection": 1, "Cryptographic Issues": 2}, + "out_of_scope_dynamic_only": 0, + "unmapped_cwe": [], + "gate": {"min_score": 80, "block_severity_reachable": 5, "blocking": false, "passed": false} } +``` + +### Consistency & Integrity + +- **Authority direction:** governance data is the source of truth; execution reads it, emits findings/signals; only the loop (via `EvolveValidator`) writes `status`/floors/score-constants back, never the pattern or the 0-5 severity. +- **Temporal/versioning:** `PatternRule.version` and `rule_policy.json.round` give per-rule audit; the journal records each lever edit; `fill_gating(reverted)` restores byte-for-byte. +- **Serialization (resume-safe):** all scan slots (`sast.targets`, `sast.findings`, `sast.rankings`, verifications) use plain dicts/dataclasses with `_target_` registration so they survive `wake()`. + +## Error Handling + +### Error Strategy + +| Error type | Condition | Response | +|---|---|---| +| **Startup config error** | An `enabled` rule's `cwe` is unmapped in PolicyStore | `assert_rules_resolve` raises; **fail loud**, abort scan before any work (Req 2.7) | +| **Malformed ruleset/policy TOML** | `tomllib` parse failure | Fail loud at load; CI catches before merge | +| **Loop edit violates bounds** | Pattern edit / severity edit / enabled→disabled / duplicate `rule_id` / unmapped CWE | `EvolveValidator` raises `StrictValidationError(kind="rule_change")`; entire round rejected (Req 3.8) | +| **Reverted re-proposal w/o rationale** | Novelty gate hit | `journal.check_novelty` blocks; loop continues with other candidates (Req 3.6) | +| **Replay smoke gate failure** | Edited ruleset/policy won't boot | Round rejected before benchmark re-run (Req 3.7) | +| **Acceptance-gate failure** | recall<90% OR FP≥10% OR score regressed OR unpredicted regression | `fill_gating(reverted)` restores config byte-for-byte (Req 4.5) | +| **HarnessX capability missing at runtime** | Extra absent or sub-harness unavailable | Stage falls back to existing equivalent; record degradation in manifest (Req 1.8) | +| **Slot contract violation** | Stage reads/writes an undeclared slot | `SlotContractMixin` raises under `OUS_SLOT_CONTRACT=strict`; warns under `warn` (Req 1.5) | +| **Slot serialization loss on resume** | Non-`_target_` slot restores as `None` | Per-slot-type round-trip test guards; `_target_` registration required | + +### Monitoring + +- `EpisodeMetricsProcessor.episode_summary` → manifest metrics block; `OTelProcessor` OTLP export when configured. +- `HarnessJournal` writes per-run `{run_id}.jsonl` / `_trace.jsonl` / `_state.json`, rotated on `SegmentBoundaryEvent`. +- The score artifact `unmapped_cwe` list surfaces policy-completeness drift; the journal changeset table surfaces every rule/policy lever edit per round. + +## Testing Strategy + +### Unit Tests +- `load_policy()` parses the `"Flaw Severity "` trailing-space header and yields 167 `CwePolicy` entries with correct `static`/`dynamic` flags. +- `project_score()` boundary cases: P=0 → 100; one severity-5 reachable finding → ~43; one low finding → ~97; shadow findings excluded. +- `resolve_severity()` returns policy severity (not the legacy rule string) and is identical for two rules sharing a CWE. +- `EvolveValidator._rules` rejects: pattern edit, severity edit, `enabled→disabled` jump, duplicate `rule_id`, unmapped CWE. +- `finding_penalty()` returns 0.0 for dynamic-only and unmapped CWEs. + +### Integration Tests +- **Byte-identical baseline (Req 6.2):** with the default ruleset and an empty `rule_policy.json`, `benchmark.py` output matches the pre-feature golden corpus exactly. +- **Startup invariant (Req 2.7):** an enabled rule with an unmapped CWE aborts the scan with a loud error. +- **Self-improvement round (Req 4.4/4.5):** a deliberately noisy rule is auto-shadowed when it drags precision; a round that regresses recall is reverted byte-for-byte. +- **Capability fallback (Req 1.8):** extra absent → `quick`/`standard` run through the sync driver unchanged; degradation recorded when a HX stage is requested but unavailable. +- **Slot round-trip (Req on resume):** each scan slot type serializes and restores without loss after `wake()`. + +### E2E / Gate Tests +- **CI detection gate (Req 6.1):** the pipeline fails the merge if recall<90% or FP≥10% on the corpus. +- **Two-condition score gate (Req 5.7/5.8):** hard gate fails on a sev5+reachable finding; score gate is advisory until `K`/`MIN_SCORE` are calibrated, then blocking. +- **Zero-dep guard (Req 7.5):** with HarnessX absent, governance/scoring/benchmark planes operate and CI gating runs in the default install. + +### Performance / Tolerance +- **HX tolerance (Req 6.4):** an enabled HarnessX-backed stage keeps recall/FP within the configured tolerance of the zero-dep baseline. +- Hunter-loop cost/token budgets enforced by `CostGuardProcessor`/`TokenBudgetProcessor`. + +## Security Considerations + +- **Supply-chain posture (Req 7):** core `dependencies = []` is preserved; HarnessX ships only as a SHA-pinned optional extra, lazy-imported behind `_has_harnessx()`. A `pip install --dry-run` audit runs before Phase 3; if the realized transitive graph pulls browser binaries / Docker SDK / a FastAPI web stack as *mandatory* deps and maintainers judge it unacceptable for a supply-chain auditing tool, the pre-specified **vendored lean subset** (Option C) is switched in under the same extra name — a packaging change, not a redesign. Submodule init is **non-recursive** so the large `verl` RL fork is never pulled. +- **Loop containment:** the meta-agent can never invent or rewrite detectors (no pattern edits) or rewrite the severity authority (no 0-5 edits); `shadow`-first staging, novelty/evidence/replay gates, and byte-for-byte revert bound its blast radius — directly preventing the "too influential" defect from re-entering via the loop. +- **Gate integrity:** the recall/FP gate is a hard constraint that is never folded into the score reward, so the loop cannot "improve" the number by shadowing rules that hurt real security. + +## Migration Strategy + +Every phase is independently shippable, test-gated, and reversible. Phases 0-2 deliver governance value at **zero dependency cost** and stay byte-identical by default. Phases 3-4 are the HarnessX adoption behind the extra. Phase 5 is the optional native end-state. + +```mermaid +flowchart LR + P0["Phase 0
Policy + Score
zero-dep, gate advisory"] --> P1["Phase 1
Rules-as-data
zero-dep, byte-identical"] + P1 --> P2["Phase 2
Benchmark->signal + ledger
zero-dep"] + P2 --> GATE{"pip --dry-run audit
(D2 decision)"} + GATE -->|graph acceptable| P3B["Phase 3
[harnessx] extra (Option B)
hunter loop + verify"] + GATE -->|graph unacceptable| P3C["Phase 3
vendored lean subset (Option C)
same extra name"] + P3B --> P4["Phase 4
Self-improvement loop live"] + P3C --> P4 + P4 --> P5["Phase 5 (optional)
full single-step harness + RL
retire HarnessRuntime"] +``` + +| Phase | Scope | Dep cost | Merge gate | Reversibility | +|---|---|---|---|---| +| **0 — Policy & score** | Vendor `CWE_Score.tsv`; `policy/verycode.py` + `scoring/project_score.py`; CWE-120→121 remap; startup CWE-resolution invariant; emit score artifact + advisory gate. `HarnessRuntime`/`PATTERN_RULES` unchanged. | **zero** | benchmark byte-identical; score is additive output | gate disable flag | +| **1 — Rules as data** | Extract `PATTERN_RULES`→`ruleset//*.toml`; delete `PatternRule.severity`; `load_ruleset()`; `resolve_severity`; add `status` + emit thresholds. | **zero** | benchmark **byte-identical** to Phase 0 | keep `PATTERN_RULES` fallback | +| **2 — Benchmark→signal + ledger** | `rule_id` on misses/FPs; per-rule precision/recall; emit `rule_policy_delta`; `rule_policy.json` ledger; calibration writes status flips re-keyed on `rule_id`. | **zero** | empty ledger == current behavior | empty ledger = no-op | +| **3 — HX execution adoption** | Run `pip --dry-run` audit (D2 gate). Add `[harnessx]` extra (pinned SHA, lazy, zero-dep CI guard); `build_sast()` + `HxScanOrchestrator`; hunter pool on real `Harness.run()`; `VerifyProcessor`; `PolicyScoring`/`ScoreProcessor`. `HarnessRuntime` stays. | **opt-in heavy** | extra-absent: unchanged; extra-present: recall/FP within tolerance | feature flag off | +| **4 — Self-improvement live** | Vendor/patch the meta-harness seam; orchestrator benchmark→delta→signal→`evolve()`→re-benchmark→accept/revert. | **opt-in** | ≥90% recall / <10% FP per-round; auto-revert | don't run `evolve` → ledger-only changes | +| **5 — Full harness + RL (optional)** | Migrate deterministic stages to single-step harness + `SlotContractMixin`; retire `HarnessRuntime`; promote score gate to blocking; optional `rl/`. | **opt-in** | golden-diff vs Phase 3 empty; gate held | per-stage revert | + +**Rollback triggers.** Any phase that drops recall<90% or raises FP≥10% on the corpus blocks the merge; any self-improvement round failing the acceptance gate reverts byte-for-byte; an unacceptable `pip --dry-run` graph switches Phase 3 to the vendored subset. diff --git a/.kiro/specs/harnessx-self-improving-rulesets/requirements.md b/.kiro/specs/harnessx-self-improving-rulesets/requirements.md new file mode 100644 index 0000000..c573d4d --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/requirements.md @@ -0,0 +1,115 @@ +# Requirements Document + +## Introduction + +OpenUltraSAST today fuses three responsibilities into one regex layer: detection patterns also carry their own severity strings and act as the sole volume control, so a noisy rule can dominate scan output, set its own criticality, and fire forever with no central governance. This feature splits those responsibilities into three governed owners — rules detect, a central CWE policy decides severity and scope, and a single project score becomes the optimization target — and adopts the HarnessX runtime exactly where it pays off: the LLM hunter loop, the LLM-judge verifier, and a meta-harness self-improvement loop that adapts rules and policy gates under hard bounds. + +The work spans five user asks. First, OpenUltraSAST shall adopt HarnessX directly at the hunter loop and migrate the existing scan flow stage-by-stage onto HarnessX processors. Second, detection rules shall become versioned data governed centrally, with severity sourced from the verycode CWE policy rather than from the rule itself. Third, the HarnessX meta-harness shall adapt rules and policy gates within bounded levers. Fourth, benchmark results shall drive that adaptation as actionable signals with a hard acceptance gate. Fifth, a Veracode-style project score shall be computed from policy-weighted, reachability-adjusted findings and gated in CI. + +Two cross-cutting constraints govern every part of this feature. The existing detection gate of at least 90% recall and under 10% false-positive rate on the benchmark corpus must never regress, at any phase, for any change. And the HarnessX runtime must be adopted in a way that preserves the project's zero-dependency posture for the governance and scoring planes, so an explicit dependency-weight decision controls whether HarnessX ships as an optional extra or as a vendored lean subset. + +## Boundary Context + +- **In scope**: detection rules as versioned data; a central CWE-to-severity policy with static/dynamic scope; a single project score (0-100) with a two-condition CI gate; HarnessX adoption at the hunter loop, the verifier, and the self-improvement meta-harness; benchmark-to-signal conversion with per-rule attribution; bounded loop authority over rule status, evidence floors, thresholds, and score constants; a dependency-weight audit and decision; preservation of the recall/FP detection gate. +- **Out of scope**: authoring new detection pattern text by the loop (human pull-request only); editing the authoritative 0-5 CWE severity by the loop (verycode-upstream authority); a full single-step HarnessX rewrite of all deterministic stages and weight-level reinforcement learning (deferred, optional, later phase); changing the reachability vocabulary, which remains exactly `reachable | inferred-file-surface | unknown`; dynamic/DAST scoring, which is report-only for a static tool. +- **Adjacent expectations**: the governance and scoring planes shall run with no HarnessX present and `dependencies = []`; CI gating (score plus benchmark) shall run in the zero-dependency default install; HarnessX, when present, shall be confined behind a capability check and lazy imports so heavy code paths stay cold in the default install. + +## Requirements + +### Requirement 1: HarnessX Direct Adoption and Scan-Flow Migration + +**Objective:** As an OpenUltraSAST maintainer, I want the scan flow to adopt the HarnessX runtime directly at the LLM hunter loop and migrate the existing stages onto HarnessX processors in a staged way, so that the project gains the HarnessX agent loop, journal, and trajectory machinery without a high-risk big-bang rewrite. + +#### Acceptance Criteria +1. Where the HarnessX extra is installed, the Scan Orchestrator shall run the hunter pool stage on a real HarnessX `Harness.run()` loop with the HarnessX control, tool, and observability processors registered in the hunter sub-harness. +2. Where the HarnessX extra is installed, the Verify Processor shall produce verification verdicts using the HarnessX llm-judge helpers and shall retain the structural verifier as a zero-dependency fallback. +3. While the HarnessX extra is absent, the Scan Runtime shall execute `quick` and `standard` scans through the existing sync driver with behavior unchanged from the pre-adoption baseline. +4. When the hunter pool stage runs under HarnessX, the Scan Orchestrator shall confine all asynchronous execution to that single stage so that the deterministic stages remain synchronous. +5. The Scan Runtime shall preserve each stage's declared read and write slot allow-list and shall validate slot access on every lifecycle hook, so that the existing provenance and audit discipline is retained under the HarnessX runtime. +6. Where a deterministic stage runs under HarnessX, the Scan Runtime shall execute it with model invocation disabled so that no deterministic stage incurs a model call. +7. When migrating event names, the Scan Runtime shall map the existing `stage_start`/`stage_end` to the HarnessX step-level hooks and `scan_id` to the HarnessX run identifier without losing trace continuity. +8. If a HarnessX-backed stage is enabled and the corresponding HarnessX capability is unavailable at runtime, then the Scan Runtime shall fall back to the existing equivalent stage and shall record the degradation in the scan manifest. + +### Requirement 2: Central Security-Affine Ruleset Governance + +**Objective:** As a security policy owner, I want detection rules to be versioned data whose severity and scope come from a central CWE policy rather than from the rule itself, so that one CWE maps to one governed severity and no rule can be too influential by setting its own criticality or volume. + +#### Acceptance Criteria +1. The Ruleset Store shall load detection rules from versioned data files in which each rule declares a pattern, a CWE, tags, and a loop-controlled `status`, and shall not carry any rule-local severity field. +2. When resolving the severity of a finding, the Policy Scoring Processor shall read severity exclusively from the Policy Store keyed on the rule's CWE and shall discard any severity string carried in legacy rule data. +3. While two enabled rules map to the same CWE, the Policy Scoring Processor shall assign both findings the identical policy-governed severity so that two rules for one CWE can no longer disagree. +4. The Ruleset Store shall support exactly three rule statuses — `enabled`, `shadow`, and `disabled` — where `enabled` fires and scores and reports, `shadow` fires and records outcomes but is excluded from the report and the score, and `disabled` does not fire. +5. When a finding's rule status is `shadow`, the Policy Scoring Processor shall exclude that finding from the reported findings and from the project-score input while still recording its outcome for precision tracking. +6. The Policy Store shall expose, per CWE, an authoritative 0-5 severity plus a static scope flag and a dynamic scope flag, and shall mark dynamic-only CWEs as report-only and never scored by the static tool. +7. If an enabled rule's CWE does not resolve in the Policy Store at startup, then the Scan Runtime shall fail loud and shall not proceed with the scan. +8. The Ruleset Store shall remap the C/C++ buffer-handling rules that reference the unmapped CWE-120 to CWE-121 (severity 5) so that those rules contribute a nonzero penalty to the project score. +9. The Ruleset Store and the Policy Store shall be loaded with the standard library only and shall require no third-party dependency. + +### Requirement 3: Bounded Rule and Policy Adaptation in the Self-Improvement Loop + +**Objective:** As an OpenUltraSAST maintainer, I want the HarnessX meta-harness to adapt rules and policy gates only through two bounded levers, so that the loop can suppress, stage, or raise the bar on noisy rules without ever inventing detectors or rewriting the severity authority. + +#### Acceptance Criteria +1. The Self-Improvement Loop shall expose exactly two new edit levers: a `rule` lever covering rule status, evidence floor, and threshold, and a `policy` lever covering evidence floor, scope, and the score constants `K` and `MIN_SCORE`. +2. The Rule Edit Validator shall reject any loop edit that modifies detection pattern text, so that pattern authoring remains human-pull-request only. +3. The Policy Edit Validator shall reject any loop edit that modifies the authoritative 0-5 CWE severity, so that the severity authority remains upstream-governed. +4. If the loop proposes moving a rule directly from `enabled` to `disabled`, then the Rule Edit Validator shall reject the edit and require `shadow` as a mandatory intermediate staging state. +5. When validating a proposed edit, the Rule Edit Validator shall assert that the edit parses, that `status` is one of `enabled`/`shadow`/`disabled`, that any threshold or evidence-floor tightening or loosening stays within configured bounds, that an added rule does not duplicate an existing rule identifier, and that every referenced CWE resolves in the Policy Store. +6. If the loop re-proposes an edit that was previously reverted without supplying a retry rationale, then the Self-Improvement Loop shall block the re-proposal via the novelty gate. +7. Before accepting any round, the Self-Improvement Loop shall run a replay smoke gate that proves the edited ruleset and policy still load and boot. +8. When any loop edit is rejected during validation, the Self-Improvement Loop shall raise a rule-change validation error and reject the entire round. +9. The Self-Improvement Loop shall record each rule and policy edit, its lever, and its outcome in the journal so that every change is attributable and reversible. + +### Requirement 4: Benchmark-Driven Iteration with a Hard Acceptance Gate + +**Objective:** As an OpenUltraSAST maintainer, I want benchmark results to become actionable per-rule signals that drive the self-improvement loop under a hard acceptance gate, so that detection quality improves automatically while regressions are reverted byte-for-byte. + +#### Acceptance Criteria +1. The Benchmark Harness shall attribute every recorded miss and every recorded false positive to a `rule_id` and shall aggregate per-rule precision and recall in its metrics. +2. The Benchmark Harness shall emit a per-rule recommendation artifact alongside its records, recommending enable, disable, shadow, threshold, or evidence-floor changes per rule. +3. When the orchestrator converts benchmark output to loop input, it shall write recall-side items as miss signals and precision-side items as verifier-reject or false-positive signals into the loop's signal file. +4. After the meta-agent edits rules or policy, the Self-Improvement Loop shall re-run the benchmark and shall accept the round only if aggregate recall is at least 90% AND aggregate false-positive rate is under 10% AND the project score did not regress on the held set AND no unpredicted regression occurred. +5. If a re-run round fails any acceptance condition, then the Self-Improvement Loop shall revert the round and restore the rule and policy configuration byte-for-byte. +6. If a rule drags aggregate precision below the gate, then the Self-Improvement Loop shall auto-shadow that rule rather than delete it, so that recall is preserved while the false-positive cost is removed. +7. When a coverage gap surfaces as a persistent miss, the Self-Improvement Loop shall nominate a new or loosened rule through the signal file while leaving pattern authoring to a human pull request. +8. The Self-Improvement Loop shall treat maximizing the project score as the optimization objective and the recall/FP gate as a separate hard feasibility constraint, so that the loop cannot improve the score by shadowing rules in a way that harms real security. + +### Requirement 5: Project Scores via verycode Policies + +**Objective:** As a security reviewer, I want a single 0-100 project score computed from policy-weighted, reachability-adjusted findings and gated in CI, so that a scan produces a bounded AppSec quality signal alongside its findings. + +#### Acceptance Criteria +1. The Score Model shall compute a single 0-100 project score from policy-governed severity weights multiplied by a reachability multiplier over the reported findings, using the standard library only. +2. The Score Model shall use reachability multipliers keyed exactly on the three emitted reachability values `reachable`, `inferred-file-surface`, and `unknown`. +3. When a finding maps to a CWE that is dynamic-only or unmapped in the Policy Store, the Score Model shall assign that finding a zero score penalty and shall report it as out-of-scope or unmapped rather than scoring it. +4. The Score Model shall exclude findings whose rule status is `shadow` from the score input before computing the project score. +5. The Score Model shall emit a scan-level score artifact containing the project score, the maximum severity, the total penalty, a by-category breakdown, the out-of-scope dynamic-only count, the unmapped CWE list, and the gate verdict, and shall include the score block in the scan manifest. +6. When confirming a false positive, the Calibration subsystem shall lower the finding's effective reachability multiplier rather than delete the rule, so that recall stays high while the score stops over-penalizing. +7. If any reported finding maps to policy severity 5 AND has `reachability_status` equal to `reachable`, then the CI Score Gate shall fail the build. +8. Where the score gate is configured as blocking, the CI Score Gate shall fail the build when the project score is below `MIN_SCORE`; until the score constants are calibrated against the corpus, the CI Score Gate shall compute and report the score advisory-first without blocking. +9. The Score Model and the CI Score Gate shall run in the zero-dependency default install with no HarnessX present. + +### Requirement 6: Detection Gate Non-Regression + +**Objective:** As an OpenUltraSAST maintainer, I want the existing detection gate of at least 90% recall and under 10% false positives on the benchmark corpus to hold across every phase and every change, so that no governance, scoring, runtime, or self-improvement change can silently degrade detection quality. + +#### Acceptance Criteria +1. The CI Pipeline shall evaluate aggregate recall and aggregate false-positive rate on the benchmark corpus on every merge and shall fail the merge if recall is below 90% or the false-positive rate is at or above 10%. +2. While the default ruleset and an empty loop ledger are in effect, the Benchmark Harness shall produce results byte-identical to the pre-feature baseline for the governance and rules-as-data phases. +3. When the self-improvement loop proposes any rule or policy change, the Self-Improvement Loop shall require the recall/FP gate to pass on the re-benchmarked candidate before the change can be accepted. +4. If a HarnessX-backed stage is enabled, then the CI Pipeline shall confirm recall and false-positive rate stay within the configured tolerance of the equivalent zero-dependency baseline. +5. The CI Pipeline shall treat the recall/FP gate as a hard constraint that is never folded into the project-score reward, so that improving the score can never substitute for meeting the detection gate. +6. If any change would reduce recall below 90% or raise the false-positive rate to 10% or above, then the CI Pipeline shall block the change regardless of any project-score improvement. + +### Requirement 7: HarnessX Dependency-Weight Decision + +**Objective:** As an OpenUltraSAST maintainer, I want an explicit dependency-weight decision that keeps the core install zero-dependency and chooses between an optional HarnessX extra and a vendored lean subset based on an audited dependency graph, so that a supply-chain auditing tool never silently inherits an unacceptable transitive dependency footprint. + +#### Acceptance Criteria +1. The Packaging Configuration shall keep the core `dependencies` list empty and shall expose HarnessX only as an optional extra pinned to a specific commit revision. +2. The Runtime shall guard every HarnessX import behind a capability check and shall import HarnessX lazily so that heavy code paths stay cold in the default install. +3. Before adopting the HarnessX execution stages, the maintainer shall run a dependency dry-run audit and shall record the realized transitive dependency graph. +4. If the audited transitive graph pulls a browser-binary, container-SDK, or web-server stack as a mandatory dependency and the maintainers judge it unacceptable for a supply-chain auditing tool, then the Packaging Configuration shall switch to a vendored lean subset of HarnessX under the same extra name with the heavy subtrees removed. +5. While HarnessX is absent, the CI Pipeline shall run a zero-dependency guard test confirming the governance, scoring, and benchmark planes operate with no HarnessX present. +6. Where the project is developed via a submodule, the Runtime setup shall initialize HarnessX non-recursively so that the large reinforcement-learning fork subtree is not pulled. +7. The Packaging Configuration shall keep the vendored lean subset pre-specified so that switching from the optional extra to the vendored subset is a packaging change and not a redesign. diff --git a/.kiro/specs/harnessx-self-improving-rulesets/research.md b/.kiro/specs/harnessx-self-improving-rulesets/research.md new file mode 100644 index 0000000..bb859d5 --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/research.md @@ -0,0 +1,191 @@ +# Research & Design Decisions + +--- +**Purpose**: Capture the discovery findings and architectural investigation behind the `harnessx-self-improving-rulesets` spec. This is the background-notes companion to `design.md` — it records *what was read*, *what was found in the source*, and *why* the three resolved decisions came out the way they did. `design.md` carries the committed architecture; this file carries the evidence and the rejected alternatives. +--- + +## Summary +- **Feature**: `harnessx-self-improving-rulesets` +- **Discovery Scope**: Complex Integration (adopt an external agent-runtime library + re-architect the in-house rule/policy/scoring planes) +- **Key Findings**: + - HarnessX and OpenUltraSAST (OUS) already both have a thing called a "harness" with "processors" and "events", but they are **incompatible at every load-bearing point**: HarnessX is async-generator / message-window-contract / transcript-state; OUS is sync / declared-`reads`/`writes`-contract / dict-state. There is no drop-in. + - HarnessX's real, reusable value for OUS is concentrated in three places: the **LLM hunter agent loop** (control/tools/observability processors), the **llm-judge verifier** (`build_judge_prompt`/`parse_judge_response`/`_call_judge`), and the **meta-harness self-improvement engine** (journal / novelty / evidence / replay / attribution). Everything else (lab web UI, browser tools, sandboxes, office skills) is unused weight. + - OUS today fuses **detector + severity authority + volume control** into one regex layer (`PATTERN_RULES` in `findings.py`); its self-improvement loop only demotes *ranking* (floored at `0.1`, keyed on `tags[0]`), never the rules; and its benchmark already names the ruleset as the thing to fix (`next_improvement_candidate="rules_or_language_hunter"`) but writes it to an inert JSON. The signals exist; the loop back onto the rules does not. + - The `verycode` policy data (`CWE_Score.tsv`, 167 CWEs, severity 0-5 + static/dynamic scope) is **complete and authoritative**, but the `verycode-score` scoring *algorithm was never implemented* (`// scorer code here` placeholder). The score formula in the design is a labeled, defensible reconstruction. + - OUS today has **zero runtime dependencies** (`dependencies = []`); HarnessX's `dependencies` list is ~30 heavyweight packages (Playwright, Docker SDK, FastAPI/uvicorn, multi-LLM SDKs, office-doc suite) with **no lean extra** to pip-install a subset — a direct hard dependency would invert the project's supply-chain posture for a security tool. + +## Research Log + +### HarnessX core composition & runtime model +- **Context**: We need to know whether OUS's `harness.HarnessRuntime` can be replaced by, or interoperate with, HarnessX's `Harness`, and how an external project builds and runs a HarnessX harness. +- **Sources Consulted**: `/tmp/HarnessX/harnessx/core/{harness,builder,processor,events,runloop,state,contract_check,config_schema,config_store,model_config,trajectory}.py` (read in full); `/home/mc/Source/OpenUltraSAST/src/openultrasast/harness.py`. +- **Findings**: + - Public composition is three objects + one method: `HarnessBuilder.build() -> HarnessConfig`; `ModelConfig.agentic(HarnessConfig) -> Harness`; `await Harness.run(task) -> HarnessResult`. **`HarnessConfig` carries NO model info** — model is always supplied separately via `ModelConfig` (key→provider map, `main` required, non-`main` keys auto-get sub-harnesses). + - `HarnessBuilder` is an immutable fluent factory: `add(proc)`, `slot(**kw)`, `add_tool`, `plugin(source)`, `__or__`/`merge` (collects conflicts → `HarnessConflictError`), `build()` (topo-sorts processors per hook by `_order` then Kahn within a bucket by `_after`). Declarative alt: `build_from_config(dict)` from a flat `processors:` list of `_target_` dicts. + - The run loop (`run_loop`) is a fixed async agent loop: `task_start` → `while True:` (step: `step_start` → `before_model` → call provider → `after_model` → per-tool `before_tool`/exec/`after_tool` → `step_end`) → `task_end`. `get_procs(key)` runs `"*"` (MultiHook) processors on every hook, then hook-specific lists. + - State is **two message tracks** (`raw_messages` append-only, `messages` effective context) + typed `slots: dict[str, StateSlot]` (the nearest analogue to OUS's flat `state` dict, but typed and `_target_`-serializable). `StatefulTrajectory` is a first-class product (`s_t = (z_t, Δz_t, a_t, o_t, e_t, r_t)`), built inline during `run_loop`, with `backfill_rewards` and `to_training_records`/`to_rl_records`. + - Hook-name alignment vs OUS: 5/8 match exactly (`task_start`, `before_model`, `after_model`, `before_tool`, `after_tool`, `task_end`); the gap is `stage_start`/`stage_end` ↔ `step_start`/`step_end`. Semantically an OUS "stage" is a coarse pipeline phase wrapping a callback (`run_stage(name, callback)`), whereas a HarnessX "step" is one model-call iteration — not the same concept. +- **Implications**: A whole-pipeline swap is high risk. The cheapest correct adoption confines HarnessX's async loop to the one OUS stage that *is* an agent loop (the hunter pool) and keeps the sync driver for deterministic stages. This directly motivates Decision D1 (phased seam, not big-bang). + +### The two incompatible processor contracts +- **Context**: OUS's defining feature is a declared `reads`/`writes` per-key state contract. We must know if HarnessX preserves it. +- **Sources Consulted**: `/tmp/HarnessX/harnessx/core/{processor,contract_check}.py`; `/home/mc/Source/OpenUltraSAST/src/openultrasast/harness.py:49-118`. +- **Findings**: + - HarnessX `Processor` is `async def process(event) -> AsyncIterator[Event]` — an async generator that yields transformed events. The recommended base is `MultiHookProcessor` with `on_*` hook methods, an `@on(EventClass)` decorator, and class-attr ordering hints (`_order`, `_after`, `_singleton_group`, `_hook`). There is **no `spec`, no `handles`, no `reads`/`writes`.** + - HarnessX's contract governs **message-window mutation**, not state keys: enforced in `ProcessorChain.process` via `_validate_messages_contract` (system stays at index 0 on `step_start`; `before_model` net length ∈ {0,+1}; appended message must be role `user`; no out-of-window edits) and per-chain `check_post_hook_invariants`. Mode is the env var `HARNESSX_CONTRACT_MODE ∈ {warn(default), strict}` — not a constructor arg. + - OUS `Processor` is **synchronous**, `def handle(event, state: dict) -> dict | None`, with `spec: ProcessorSpec(name, version, handles, reads, writes)`; `_run_processor` enforces *state-key* discipline (reads must pre-exist; writes must be declared; return must be a dict). `ContractMode` is a `"strict"|"warn"` constructor arg. +- **Implications**: The two contracts do not map onto each other. OUS's per-key allow-list (its provenance/audit discipline) has **no home in HarnessX**. To keep that discipline we must reimplement it as a `SlotContractMixin` on our HarnessX processors (`reads_slots`/`writes_slots` class attrs validated in a base `on_*` wrapper), gated by our own `OUS_SLOT_CONTRACT` env. This is Decision D7. + +### HarnessX built-in processors — reuse map +- **Context**: Which HarnessX processors can OUS register directly, which are reusable as patterns/helpers, and which are irrelevant? +- **Sources Consulted**: `/tmp/HarnessX/harnessx/processors/` (control, tools, multi_model, observability, evaluation, memory, context); `/home/mc/Source/OpenUltraSAST/src/openultrasast/cli.py` (`_run_scan`). +- **Findings**: + - **Register-direct only inside the hunter loop** (the one real agent loop): `LoopDetectionProcessor`, `CostGuardProcessor`/`TokenBudgetProcessor`, `ToolFailureGuard`, `SelfVerifyProcessor`, `ToolFilterProcessor`/`ToolWhitelistProcessor`, `ModelRouterProcessor`, `EpisodeMetricsProcessor`/`OTelProcessor`/`CheckpointProcessor`. + - **Reuse as helper functions** for `verify`: the `llm_judge` code (`build_judge_prompt`, `parse_judge_response` with the verdict whitelist, the two-attempt repair loop `_call_judge`) plus the strict-JSON verdict schema (`verdict/confidence/cause/missing_capability`/`lesson`/`missing`). The judge is explicitly told "ground truth is intentionally withheld" — matching OUS's no-oracle scanning. + - **Reuse as pattern** for `calibrate`/`record_calibration`: `MemoryRetrievalProcessor` + `MemoryPolicy` (retrieve/add, should_retrieve/should_store) is the clean interface to refactor the FP ledger toward; `MemoryExtractionProcessor` mirrors distilling verifier outcomes into durable learnings. Storage stays a custom JSON ledger. + - **Keep custom** for every SAST-domain stage: `static_mapping`, `preprocess`, `entry_point_mapping`, `quick_findings`, `report`, `sarif`, `manifest`, and the benchmark recall/FP scoring (HarnessX `EvalResult.score` is a single float — too coarse for the recall/FP gate). +- **Implications**: Adoption is bimodal — register HarnessX classes directly only in the hunter sub-harness; everywhere else port logic as helpers or patterns. This is the backbone of the §3 flow-migration table in `design.md`. + +### HarnessX self-improvement (meta_harness) and the rule-adaptation seam +- **Context**: Can an external signal (a verifier rejection or a benchmark miss) drive a *detection-rule* change through HarnessX's self-improvement loop, and how does it compare to OUS's current calibration? +- **Sources Consulted**: `/tmp/HarnessX/harnessx/meta_harness/{agent,journal,replay,validate_workflow}.py`, `processors/contract_autocheck.py`, `workers/trajectory_digester.py`, `rl/{builder,config,task}.py`; `/home/mc/Source/OpenUltraSAST/src/openultrasast/{calibration,benchmark,findings,rank,cli}.py`. +- **Findings**: + - HarnessX self-improves a *harness* round by round. The unit of change is a `HarnessConfig` (the meta-agent literally rewrites the inner agent's config/processors/templates as files); the unit of memory is a markdown **journal** with per-`## Round N` YAML frontmatter; the unit of safety is a multi-phase **validation gate**. + - The journal classifies every change by a **lever** from `_VALID_LEVERS = {configuration, control, action, instruction}` and tracks `predicted_affected` task_ids plus a `gating_attribution` (`flipped/still_F/regressed/still_T/absent`). `build_context` renders a per-lever **time-decayed Beta posterior** scoreboard (`_DECAY=0.9`) and an explicit "Reverted hypotheses — do not re-propose" list. + - `EvolveValidator.run` enforces validity (canonicalize → contract → diff → replay smoke gate), policy (novelty + evidence, only when the diff is non-empty), advisory. `check_novelty` hard-blocks re-proposing a reverted hypothesis or reverted `(levers, predicted_affected)` signature unless a `retry_rationale` is present. `_evidence` requires a `candidates.md` `## Candidate C-N` cross-referenced by the journal. `run_synthetic_task_smoke_gate` proves the evolved config boots end-to-end. `ContractAutoCheckProcessor` gives in-session `[AUTO-CHECK]` feedback on every authored `processors/*.py`/`tools/*.py` edit. + - **The concrete seam to drive a rule change**: add a lever value to `journal._VALID_LEVERS`; carry the external signal in via a `rule_signals.json` read-context artifact + `_render_task_brief` deliverable; add a validity-phase check (`EvolveValidator._rules`) that raises `StrictValidationError(kind="rule_change")`; add changeset buckets (`compute_changeset` + `journal._CHANGESET_KEYS_ORDER`); extend `contract_autocheck` to classify rule files. The novelty/evidence/replay/attribution machinery then applies for free. + - **OUS by comparison**: detection rules in `findings.py` are static and never modified. Calibration (`calibration.py`) only does `calibrated = max(0.1, calibrated - 0.5)` on a *file's* ranking priority (keyed on `tags[0]`, not `rule_id`) plus a prompt-constraint string — it never disables/tightens/adds a rule, has no novelty gate, no attribution. Benchmark misses produce a `BenchmarkCalibrationRecord` whose `next_improvement_candidate` is hard-coded to `"rules_or_language_hunter"` and written to an **inert** `calibration_records.json` that nothing consumes. The recall side is a fully open loop. +- **Implications**: OUS has exactly the *signals* HarnessX needs (verifier rejections = precision side; benchmark misses = recall side) but no loop closing back onto the rules. The meta_harness is the engine that closes it. This motivates Decision D3 (which levers, and their hard bounds) and the §5/§6 design. + +### The verycode policy & scoring model +- **Context**: The design needs a central CWE→severity authority and a project score. What does `verycode` actually provide? +- **Sources Consulted**: `/tmp/verycode-policies/{README.md,CWE_Score.tsv}`; `gh repo view norandom/verycode-score` (incl. `action.yaml`, `main.go`); `gh repo view norandom/verycode-lib`; `/home/mc/Source/OpenUltraSAST/src/openultrasast/findings.py`. +- **Findings**: + - `CWE_Score.tsv` is the real artifact: 167 data rows, one per CWE, columns `Flaw Category | CWE ID | CWE Name | Flaw Severity (0-5) | Static (X/empty) | Dynamic (X/empty)`. Severity distribution: 12×5, 15×4, 97×3, 31×2, 0×1, 12×0. Scope: 154 Static, 45 Dynamic, 13 Dynamic-only, none neither. (Header column 4 is literally `"Flaw Severity "` with a trailing space.) + - **The scoring algorithm does not exist.** `norandom/verycode-score` is the unmodified `the-gophers/go-action` "Tweeter" template with `// scorer code here` and a `--policy policies/standard.json` flag; `norandom/verycode-lib` is a one-line stub README. The data shape + the "GitHub Advanced Security findings, weighted by CWE severity" descriptions imply a Veracode-style 0-100 pipeline; the design reconstructs the formula and labels it as such. + - **Real coverage gap**: CWE-120 (Classic Buffer Overflow) is **absent from the TSV**, but 3 high-value C/C++ rules use it (`c-unsafe-copy`, `c-unsafe-memory-copy`, `c-unsafe-scanf`). Unmapped → those rules would contribute 0 to the score. CWE-121 (Stack Buffer Overflow) is present at severity 5 and is the precise classic-overflow CWE. + - **Severity mismatch**: OUS rule strings disagree with policy (rules label CWE-78/95/134 "high"; policy rates them 5/critical; rules label CWE-94/502 "high"; policy rates them 3/medium). The policy is the authoritative source. + - The OUS reachability vocabulary is exactly `reachable | inferred-file-surface | unknown` (verified in `findings.py`), which maps cleanly onto a `REACH_MULT = {reachable:1.0, inferred-file-surface:0.6, unknown:0.4}` exploitability multiplier — and that multiplier is the natural FP-calibration knob (lower effective reachability instead of deleting a rule). +- **Implications**: Severity must flow from PolicyStore (`CWE_Score.tsv`), not from the rule. Day-one fixes: remap the 3 CWE-120 rules to CWE-121 (Decision D5); discard rule string-severities (Decision D4). The score is a stdlib-only reconstruction (Decision D8 ships it advisory-first until `K`/`MIN_SCORE` are calibrated). + +### Installability & dependency weight +- **Context**: How does an external project depend on HarnessX, and what does it cost a zero-dependency security tool? +- **Sources Consulted**: `/tmp/HarnessX/pyproject.toml`, `harnessx/__init__.py`, `core/builder.py`, `plugins/{base,loader,discovery,convert}.py`, `examples/{coding,research,assistant}/harness.py`, `extensions/plugins/workflow/plugin.py`, `.gitmodules`; `/home/mc/Source/OpenUltraSAST/pyproject.toml`. +- **Findings**: + - HarnessX is **library-first** and **effectively not on PyPI** (`name="harnessx"`, `version="0.1.0"`, Beta; consumed from Git). `requires-python = ">=3.11"` — exact match with OUS, no floor conflict. Depend via a SHA-pinned Git URL (the version string won't move). + - Custom stages map cleanly onto HarnessX's plugin model: each SAST stage is a `MultiHookProcessor`; group stages + tools + skills into a `HarnessPlugin` mounted via `builder.plugin(...)`; load in-process (instance/class) rather than via directory/`plugin.json`. + - **Dependency-weight is the headline risk.** HarnessX's `dependencies` (always-installed, not extras) is ~30 heavyweight packages: `anthropic`/`openai`/`litellm`/`tiktoken`, `fastapi`/`uvicorn`/`websockets`, `playwright` (ships browser binaries), `docker`/`e2b`, `hydra-core`/`omegaconf`/`structlog`/`loguru`, `opentelemetry-*`, `mcp`, `python-pptx`/`python-docx`/`openpyxl`/`reportlab`/`pdfplumber`. **There is no lean extra** — `pip install harnessx` pulls everything. OUS currently has `dependencies = []`. + - **Mitigating fact**: HarnessX uses **pervasive lazy imports** (heavy modules imported inside functions). So the heavy deps are *installed* but only *imported on demand* — which makes both an optional-extra opt-in and a vendored subset genuinely viable. + - **Submodule caveat**: `.gitmodules` declares a nested `recipe/verl_harnessX/verl` → a large verl RL fork OUS does not need; never `--recursive` init. +- **Implications**: A hard dependency is off the table (it inverts the zero-dep stance and pins to an unmovable Beta). Options narrow to (B) optional extra and (C) vendored lean subset. The lazy-import architecture is what makes B keep heavy code cold by default and makes C a surgical packaging switch. This is Decision D2. + +### Current OUS rule-system problem analysis +- **Context**: Establish, in code terms, why the user's complaint ("rules are too influential and not centrally managed") is true. +- **Sources Consulted**: `/home/mc/Source/OpenUltraSAST/src/openultrasast/{findings,rank,calibration,benchmark,verification,harness,config,cli}.py`. +- **Findings** (six code-level defects): + 1. **Pattern matches dominate output.** In quick mode, every non-comment regex match becomes a finding (`quick_scan_findings`); the only gate is language. Ranking only sorts; verification rejects only on `reachability_status == "unknown"`. A noisy regex sets output volume and FP rate directly — the rule *is* the result. + 2. **Severity is rule-local.** `PatternRule.severity` is a string literal copied straight onto the finding (`_finding_from_match`). Two rules for the same CWE can disagree; nothing reconciles them. + 3. **No central store.** The ruleset is a Python tuple compiled into the package; `config.py` has no rule/policy section. You cannot enable/disable/tune a rule without editing source and cutting a release. + 4. **Calibration demotes ranking, not rules.** `calibrate_ranking` only subtracts `learning.demotion` (default 0.5) from a *file's* `priority`, floored at `MIN_CALIBRATED_PRIORITY = 0.1`, keyed on `tags[0]`. A chronically wrong rule fires at full severity forever; learning targets the wrong object. + 5. **No CWE-policy linkage.** `cwe` is free-text used only for rationale text and loose benchmark matching. No table maps CWE → governed severity/enablement/evidence floor. + 6. **No project-score weighting.** Severity and emission are global constants; the ranking signal (`composite_priority`) only re-sorts. A low-severity crypto finding in a fuzzable network parser is never promoted; a high-severity finding in dead code is never demoted. +- **Implications**: The regex layer is simultaneously **detector, severity authority, and volume knob**, while ranking/verification/calibration/benchmark can only act *around* the rules, never *on* them. The fix is to split those three roles into three owners (rules detect / policy decides severity & scope / score is the optimization target) with a one-directional authority model (governance drives execution; execution emits signals; the loop edits governance under a gate). This is the organizing principle of `design.md`. + +## Architecture Pattern Evaluation + +These are the three "architect angles" that were considered for the overall shape. Each is internally coherent; the synthesis in `design.md` borrows from all three and resolves their three disagreements (D1/D2/D3). + +| Option | Description | Strengths | Risks / Limitations | Notes | +|--------|-------------|-----------|---------------------|-------| +| **Incremental seam** | Keep the OUS sync driver; run HarnessX only inside the `hunter_pool` callback via `asyncio.run`. Rules→data + policy + score added around the existing pipeline. Dependency = optional extra (B) with vendored (C) as escape hatch. Loop edits a `"rule"` lever (status/threshold). | Smallest blast radius; confines the async surface to one stage; keeps the recall/FP gate green per-merge; preserves zero-dep default. | Two harness contracts coexist for a while; `HarnessRuntime` lingers; deterministic stages don't gain HarnessX trajectories yet. | Chosen as the base trajectory (Phases 0-4). | +| **Native rewrite** | Delete `HarnessRuntime` entirely; the *whole* scan becomes a single-step HarnessX harness with `skip_model=True` + slot dataflow; vendored lean subset (C) as the committed default. Loop edits a `"rule"` lever; patterns stay human-PR'd. | Cleanest end-state; one runtime, one contract; full trajectory/journal coverage of every stage; smallest auditable dep graph. | Touches every stage at once — too much blast radius to keep the gate green per-merge; vendoring is permanent fork-maintenance burden. | Demoted to optional Phase 5+; its end-state (single-step harness, retire `HarnessRuntime`) is the long-term target. | +| **Policy-first** | The whole scan runs as processors on a fixed loop; central CWE policy is the centerpiece; loop edits `"rule"` + `"policy"` levers (policy = evidence floors/scope, *not* the 0-5 CWE severity). Dependency = optional extra (B) default, vendored (C) fallback. | Strongest governance story; two levers give the loop the right surface to "stop rules being too influential" without inventing detectors or rewriting severity. | Same whole-pipeline-at-once risk as native if taken literally; needs careful bounds so the loop can't corrupt the ruleset. | Contributed the two-lever model (D3) and the bounded-authority safety rules. | + +## Design Decisions + +The three architect angles agreed on the destination (rules→governed data; severity from a central CWE policy; a gated project score; HarnessX hosting the hunter loop and a rule-adapting self-improvement loop). They disagreed on three load-bearing choices. Each is resolved below with the recall/FP gate as the deciding tie-breaker. (D4-D9 are smaller decisions that follow.) + +### Decision D1: Scope of the HarnessX migration +- **Context**: How much of the scan moves onto HarnessX, and how fast? +- **Alternatives Considered**: + 1. Incremental seam — sync driver stays; HarnessX only in the hunter callback. + 2. Native rewrite — delete `HarnessRuntime`; whole scan becomes a single-step harness immediately. + 3. Policy-first whole-loop — whole scan as processors on a fixed loop. +- **Selected Approach**: **Phased.** Adopt HarnessX directly *at the hunter loop* first (Phase 3); migrate deterministic stages onto a single-step harness only later and optionally (Phase 5+), retiring `HarnessRuntime` only after the seam is proven and the gate has held. +- **Rationale**: The sync→async + contract mismatch is the single highest-risk item in the whole effort. The native rewrite touches every stage at once — too much blast radius to keep ≥90% recall / <10% FP green per-merge. The incremental seam ships the *actual* HarnessX value (agent loop, journal, `StatefulTrajectory`) at exactly the one stage that is already an agent loop. +- **Trade-offs**: Two harness contracts coexist through Phases 3-4 and `HarnessRuntime` lingers; in exchange every phase is independently shippable, test-gated, and reversible. +- **Follow-up**: Golden-output diff gate at each phase; confine `asyncio.run` to the hunter callback; `SlotContractMixin` to preserve key-level discipline. + +### Decision D2: Dependency strategy +- **Context**: A zero-dependency security tool considering a ~30-package agent runtime with no lean extra. +- **Alternatives Considered**: + 1. (A) Hard dependency in main deps. + 2. (B) Optional extra `openultrasast[harnessx]`, SHA-pinned, lazy-imported. + 3. (C) Vendored lean subset under `src/openultrasast/_vendor/harnessx/`. +- **Selected Approach**: **(B) is the shipping default; (C) is a pre-committed, triggered fallback** gated by an explicit `pip install --dry-run` audit before Phase 3. Core `dependencies = []` stays; all HarnessX imports are lazy behind a `_has_harnessx()` check. +- **Rationale**: HarnessX's pervasive lazy imports mean the heavy deps are *installed* but only *imported on demand*, so a clean opt-in extra already keeps heavy code paths cold in the default install — with the least maintenance burden, while tracking upstream. (A) permanently inverts the project's stance and pins to an unmovable Beta `0.1.0`. (C) adds permanent fork-maintenance and should only be paid for when the *installed* graph (not the *imported* graph) is itself unacceptable. +- **Trade-offs**: With B, enabling the extra still inherits the full HarnessX dep set (no lean upstream extra exists); every HarnessX import must be guarded. +- **Follow-up**: Run the dry-run audit and record the realized transitive graph. **Decision rule**: if it pulls Playwright browser binaries / Docker SDK / a FastAPI web stack as *mandatory* transitive deps and maintainers judge that unacceptable, switch to (C) — vendor `harnessx/{core,plugins,processors,bundles,meta_harness,workspace,tools/base}` + the `llm_judge`/`journal`/`validate_workflow`/`agent`/`contract_autocheck` modules, delete `lab/`/browser/sandbox/office subtrees, re-pin a minimal set. The lazy-import design makes the deletion surgical. Never use the recursive submodule path (`recipe/verl_harnessX/verl`). + +### Decision D3: What the self-improvement loop is allowed to edit +- **Context**: The user's core complaint is that rules are "too influential." The loop must be able to *govern* rules without becoming a new way for them to run wild. +- **Alternatives Considered**: + 1. Single `"rule"` lever (status/threshold). + 2. `"rule"` lever; patterns stay human-PR'd (native angle). + 3. `"rule"` + `"policy"` levers, where policy = evidence floors/scope, *not* CWE severity (policy-first angle). +- **Selected Approach**: **Two bounded levers.** `"rule"` = `status` (enabled/shadow/disabled) + evidence-floor + threshold, **never pattern text**. `"policy"` = evidence floor + scope + score constants `K`/`MIN_SCORE`, **never the authoritative 0-5 CWE severity**. Patterns and CWE severities are human-PR-only. +- **Rationale**: This is the precise answer to "stop rules being too influential" — the loop may *suppress / stage / raise-the-bar*, never *invent detectors* or *rewrite the severity authority*. Bounds + mandatory `shadow` staging + auto-revert keep loop authority safe; the meta_harness novelty/evidence/replay/attribution gates apply for free. +- **Trade-offs**: The loop cannot fix a bad pattern itself (it can only `shadow` it and nominate a human PR), and cannot re-weight a CWE — both are deliberate guardrails. +- **Follow-up**: Implement `EvolveValidator._rules`/`._policy` raising `StrictValidationError(kind="rule_change")`; enforce the four "may NOT" bounds (no pattern edits, no 0-5 severity edits, no `enabled→disabled` without `shadow`, no reverted re-proposal without `retry_rationale`). + +### Decision D4: Severity authority +- **Context**: Two rules for the same CWE can disagree on severity today. +- **Selected Approach**: PolicyStore (`CWE_Score.tsv`) is authoritative; `resolve_severity(policy, rule.cwe, target, ranking)` replaces the literal `rule.severity` read; `PatternRule.severity` is removed. Policy always wins; rule string-severities are discarded. +- **Rationale**: One CWE = one governed severity, reconciled centrally; ends rule disagreement. +- **Trade-offs**: Loses the per-rule severity hint (acceptable — it was an authoring accident, not a governed decision). + +### Decision D5: The CWE-120 coverage gap +- **Context**: 3 C/C++ buffer rules use CWE-120, which is absent from `CWE_Score.tsv`. +- **Selected Approach**: Remap `c-unsafe-copy`/`c-unsafe-memory-copy`/`c-unsafe-scanf` to **CWE-121** (Stack Buffer Overflow, severity 5) in the ruleset TOML. +- **Rationale**: CWE-121 is the precise classic-overflow CWE and is already at severity 5; this avoids forking the upstream policy file. Otherwise those high-value rules contribute 0 to the score. +- **Follow-up**: Add a hard CI invariant — every `enabled` rule's `cwe` must resolve in PolicyStore (fail loud at startup); log `unmapped_cwe` in the score artifact; PR genuine gaps upstream to verycode-policies. + +### Decision D6: Response to a confirmed false positive +- **Context**: How does the system react to an FP without hurting recall? +- **Selected Approach**: Lower the finding's effective **reachability multiplier** and/or `shadow` the rule — **never delete it**. +- **Rationale**: The score stops over-penalizing while the rule keeps firing (tracked, just not scored/reported), preserving ≥90% recall and pushing FP <10%. This is exactly the recall/FP balance the project gates on. + +### Decision D7: Where OUS's `reads`/`writes` contract lives +- **Context**: HarnessX has no per-key state contract; OUS's defining discipline must survive. +- **Selected Approach**: Reimplement as a `SlotContractMixin` on our `MultiHookProcessor`s (`reads_slots`/`writes_slots` class attrs validated in a base `on_*` wrapper), gated by `OUS_SLOT_CONTRACT=strict|warn`. +- **Rationale**: The two contracts are incompatible; the mixin preserves provenance/audit discipline inside HarnessX's runtime without trying to bend the message-window contract. + +### Decision D8: Project-score gate rollout +- **Context**: The verycode score formula is a reconstruction (no upstream implementation). +- **Selected Approach**: Ship the score gate **advisory-first** (computed and reported, not blocking); promote to blocking only after `K`/`MIN_SCORE` are calibrated against the benchmark corpus. Score computation is stdlib-only and always on (zero-dep), so CI gating runs in the default install with no HarnessX present. +- **Rationale**: The formula needs corpus calibration before it can safely block a merge; keeping it zero-dep keeps the CI gate independent of the optional harness. + +### Decision D9: RL adoption +- **Context**: HarnessX has an `rl/` subsystem (weight-level improvement) alongside the config-level meta_harness. +- **Selected Approach**: Optional, Phase 5+. +- **Rationale**: Config-level adaptation via the meta_harness is sufficient to close the rule/policy loop; weight-level RL over hunter trajectories is purely additive and not on the critical path. + +## Risks & Mitigations +- **Sync↔async + contract incompatibility (High)** — confine the async surface to the hunter stage via `asyncio.run` in its callback (Phase 3); `SlotContractMixin` for key-level discipline; deterministic stages run `skip_model`; golden-output diff gate at each phase. +- **Dependency posture inversion (High)** — governance/scoring planes stay zero-dep; HarnessX strictly behind the extra + lazy imports + pinned SHA; CI zero-dep guard test; `pip --dry-run` audit before Phase 3; vendored-subset fallback pre-specified. +- **Benchmark-gate regression from rule auto-tuning (High)** — the recall/FP gate is the per-round acceptance test (aggregate + per-rule, hard constraint); `shadow` before `disabled`; novelty + attribution block thrash and reverted re-proposals; auto-revert byte-for-byte. +- **Score gaming the gate (High)** — keep the gate as a *hard constraint*, never part of the reward; track unpredicted regressions; add a score-regression check to acceptance. +- **Meta-agent corrupts the ruleset, re-introducing "too influential" (Medium)** — bounded levers (D3): no pattern rewrites, no severity edits, mandatory `shadow` staging, no `enabled→disabled` jump; recall gate auto-reverts; every change journaled and novelty-checked. +- **Policy completeness / CWE drift (Medium)** — day-one CWE-120→CWE-121 remap; hard CI invariant that every enabled rule's CWE resolves; `unmapped_cwe` logged; treat verycode-policies as authoritative-upstream and PR gaps there. +- **Frozen Beta upstream (Medium)** — pin SHA; document in `VENDOR.md`; keep the meta_harness patch minimal/isolated; rule lever degrades gracefully (absent patch = no rule lever, loop still does ranking demotion); vendored subset removes upstream-instability exposure for depended modules. +- **Slot serialization loss on resume (Medium)** — all scan slots use plain dicts/dataclasses with `_target_` registration; per-slot-type round-trip serialization test (non-serializable content restores as `None` on `wake()`). +- **Two-store coordination drift (Low-Med)** — a single `resolve_severity` owns the rule-status + policy-gate combination; the changeset records both lever edits per round; the benchmark regression suite catches interaction regressions. + +## References +- HarnessX core — `/tmp/HarnessX/harnessx/core/{harness,builder,processor,events,runloop,state,contract_check,model_config,trajectory}.py` — composition/run-loop/contract/state model. +- HarnessX processors — `/tmp/HarnessX/harnessx/processors/{evaluation/llm_judge.py, evaluation/strategies/evaluators/prm.py, observability/{otel_proc,episode_metrics,checkpoint}.py, memory/memory_retrieval.py, control/loop_detection.py}` — reuse map (verify/calibrate/observability). +- HarnessX meta_harness — `/tmp/HarnessX/harnessx/meta_harness/{agent,journal,replay,validate_workflow}.py`, `processors/contract_autocheck.py`, `workers/trajectory_digester.py`; `rl/{builder,config,task}.py` — self-improvement engine + rule-adaptation seam. +- HarnessX packaging — `/tmp/HarnessX/pyproject.toml`, `harnessx/__init__.py`, `plugins/{base,loader,discovery,convert}.py`, `examples/{coding,research,assistant}/harness.py`, `extensions/plugins/workflow/plugin.py`, `.gitmodules` — installability + dependency weight. +- verycode policy — `/tmp/verycode-policies/{CWE_Score.tsv,README.md}`; `gh repo view norandom/verycode-score` (`action.yaml`, `main.go` placeholder); `gh repo view norandom/verycode-lib` (stub) — policy data (complete) vs scoring algorithm (never implemented). +- OUS current system — `/home/mc/Source/OpenUltraSAST/src/openultrasast/{findings,rank,calibration,benchmark,verification,harness,config,cli}.py` — `PATTERN_RULES`, ranking-only calibration (`MIN_CALIBRATED_PRIORITY=0.1`), inert `next_improvement_candidate`, sync `HarnessRuntime`, no rule/policy config section. +- Companion design — `.kiro/specs/harnessx-self-improving-rulesets/design.md` (committed architecture); detection gate per `MEMORY.md`: ≥90% recall / <10% FP on the benchmark corpus. diff --git a/.kiro/specs/harnessx-self-improving-rulesets/spec.json b/.kiro/specs/harnessx-self-improving-rulesets/spec.json new file mode 100644 index 0000000..14ed480 --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/spec.json @@ -0,0 +1,22 @@ +{ + "feature_name": "harnessx-self-improving-rulesets", + "created_at": "2026-06-22T10:07:18Z", + "updated_at": "2026-06-22T10:07:18Z", + "language": "en", + "phase": "tasks-generated", + "approvals": { + "requirements": { + "generated": true, + "approved": false + }, + "design": { + "generated": true, + "approved": false + }, + "tasks": { + "generated": true, + "approved": false + } + }, + "ready_for_implementation": false +} diff --git a/.kiro/specs/harnessx-self-improving-rulesets/tasks.md b/.kiro/specs/harnessx-self-improving-rulesets/tasks.md new file mode 100644 index 0000000..ca0459e --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/tasks.md @@ -0,0 +1,275 @@ +# Implementation Plan + +This plan splits the fused regex layer into three governed owners (rules detect, CWE policy decides severity/scope, project score is the optimization target) and adopts HarnessX behind an optional extra. Tasks are grouped into four phases. Each phase is independently shippable and keeps the ≥90% recall / <10% FP detection gate green. Phases 1-2 (Foundation) and the early Core work stay zero-dependency and byte-identical by default; the HarnessX execution and self-improvement work (Integration) lives behind a lazy, SHA-pinned extra; Validation locks the cross-cutting gates and the dependency-weight decision. + +## Phase 1 — Foundation: Governance & Scoring Planes (zero-dep) + +- [ ] 1. Stand up the central CWE policy authority +- [ ] 1.1 Vendor the verycode CWE severity data and build the Policy Store loader + - Vendor the verycode CWE-to-severity dataset into the governance plane and parse it with the standard library only, tolerating the trailing-space severity column header. + - Expose, per CWE, an authoritative 0-5 severity, a static scope flag, and a dynamic scope flag; mark dynamic-only CWEs as report-only. + - Observable completion: loading the policy yields the full set of CWE policy entries with correct static/dynamic flags and zero third-party imports, verified by a unit test. + - _Requirements: 2.6, 2.9_ + - _Boundary: PolicyStore_ + +- [ ] 1.2 Expose policy-governed severity resolution + - Resolve a finding's severity exclusively from the Policy Store keyed on the rule's CWE, discarding any severity string carried in legacy rule data. + - Guarantee that two findings whose rules map to the same CWE receive identical severity. + - Observable completion: a resolution call returns the policy severity (never a legacy string) and is identical for two rules sharing a CWE, proven by a unit test. + - _Requirements: 2.2, 2.3_ + - _Boundary: PolicyStore_ + +- [ ] 1.3 Enforce the fail-loud startup CWE-resolution invariant + - Assert at startup that every enabled rule's CWE resolves in the Policy Store; abort the scan loudly before any work when an enabled rule's CWE is unmapped. + - Observable completion: an enabled rule with an unmapped CWE aborts the scan with a loud error and no findings are produced, proven by an integration test. + - _Requirements: 2.7_ + - _Boundary: PolicyStore_ + - _Depends: 1.1_ + +- [ ] 2. Build the project Score Model and CI gates (zero-dep) +- [ ] 2.1 Compute the 0-100 policy-weighted, reachability-adjusted project score + - Compute a single 0-100 project score from policy-governed severity weights multiplied by a reachability multiplier over reported findings, using the standard library only. + - Key the reachability multipliers exactly on the three emitted reachability values (`reachable`, `inferred-file-surface`, `unknown`) without redefining the vocabulary. + - Assign a zero score penalty to findings whose CWE is dynamic-only or unmapped, reporting them as out-of-scope or unmapped instead of scoring them. + - Observable completion: a no-penalty corpus scores 100, a single severity-5 reachable finding scores roughly 43, and dynamic-only/unmapped findings contribute zero penalty, proven by boundary unit tests. + - _Requirements: 5.1, 5.2, 5.3, 5.9_ + - _Boundary: ScoreModel_ + - _Depends: 1.1_ + +- [ ] 2.2 Emit the scan-level score artifact and manifest score block + - Emit a score artifact containing the project score, the maximum severity, the total penalty, a by-category breakdown, the out-of-scope dynamic-only count, the unmapped-CWE list, and the gate verdict; include the score block in the scan manifest. + - Observable completion: a scan writes a score artifact with every required field and merges the score block into the manifest, verified against the documented artifact schema. + - _Requirements: 5.5_ + - _Boundary: ScoreModel_ + - _Depends: 2.1_ + +- [ ] 2.3 Implement the two-condition CI score gate (advisory-first) + - Fail the build when any reported finding maps to policy severity 5 and has reachability `reachable`. + - When configured as blocking, fail the build when the project score is below the minimum; until the score constants are calibrated against the corpus, compute and report the score advisory-first without blocking. + - Observable completion: the hard gate fails on a severity-5 reachable finding while the score gate reports advisory-only by default, proven by gate tests. + - _Requirements: 5.7, 5.8, 5.9_ + - _Boundary: CI Score Gate_ + - _Depends: 2.1_ + +- [ ] 2.4 Centralize ruleset/policy/score configuration + - Add configuration entries for the ruleset, policy, and score data locations plus the gate thresholds and emission floors used by later phases. + - Observable completion: the scan reads the score constants and emission thresholds from configuration rather than hardcoded literals, confirmed by a config-driven test. + - _Requirements: 5.8_ + - _Boundary: Packaging Configuration, ScoreModel_ + +## Phase 2 — Core: Rules-as-Data and Benchmark Signals (zero-dep, byte-identical) + +- [ ] 3. Convert detection rules into versioned, governed data +- [ ] 3.1 Externalize detection patterns into a versioned Ruleset Store + - Load detection rules from versioned data files in which each rule declares a pattern, a CWE, tags, and a loop-controlled status, carrying no rule-local severity field. + - Remove the rule-local severity field from the in-memory rule model so severity can only come from policy. + - Observable completion: the existing pattern rules load from versioned data files with no severity attribute present, proven by a unit test asserting the absence of the field. + - _Requirements: 2.1, 2.9_ + - _Boundary: RulesetStore_ + +- [ ] 3.2 Implement the three-status volume control + - Support exactly three rule statuses — `enabled`, `shadow`, `disabled` — where enabled fires/scores/reports, shadow fires and records outcomes but is excluded from report and score, and disabled does not fire. + - Exclude shadow-status findings from the reported set and from the score input while still recording their outcome for precision tracking. + - Observable completion: an enabled rule appears in report and score, a shadow rule is excluded from both yet records an outcome, and a disabled rule never fires, proven by a unit test across all three statuses. + - _Requirements: 2.4, 2.5, 5.4_ + - _Boundary: RulesetStore, ScoreModel_ + - _Depends: 3.1, 2.1_ + +- [ ] 3.3 Remap the unmapped C/C++ buffer rules to a scored CWE + - Remap the C/C++ buffer-handling rules that reference the unmapped CWE-120 to CWE-121 so they resolve in policy and contribute a nonzero penalty. + - Observable completion: the remapped buffer rules resolve to a severity-5 policy entry and produce a nonzero score penalty, verified by a unit test. + - _Requirements: 2.8_ + - _Boundary: RulesetStore_ + - _Depends: 3.1, 1.1_ + +- [ ] 3.4 Wire findings and severity resolution through the governance plane + - Pass the loaded ruleset into the finding-detection stage instead of reading an implicit module-level rule list, and resolve every finding's severity through the Policy Store. + - Apply emission thresholds so not every regex match becomes a reported finding, replacing the old "rule is its own volume control" behavior. + - Observable completion: a scan produces findings with policy-resolved severity and shadow findings filtered out, while the default ruleset keeps benchmark output byte-identical to the pre-feature baseline. + - _Requirements: 2.2, 2.3, 2.5, 6.2_ + - _Boundary: PolicyScoringProcessor, RulesetStore, PolicyStore_ + - _Depends: 3.1, 3.2, 1.2_ + +- [ ] 4. Make benchmark output an actionable per-rule signal +- [ ] 4.1 Attribute misses and false positives to rules + - Attribute every recorded miss and every recorded false positive to a rule identifier and aggregate per-rule precision and recall in the benchmark metrics. + - Observable completion: a benchmark run reports per-rule precision/recall and every miss and false positive carries a rule identifier, verified by a metrics test. + - _Requirements: 4.1_ + - _Boundary: Benchmark Harness_ + - _Depends: 3.1_ + +- [ ] 4.2 Emit a per-rule recommendation delta artifact + - Replace the inert improvement-candidate stub with a per-rule recommendation artifact recommending enable, disable, shadow, threshold, or evidence-floor changes per rule. + - Observable completion: a benchmark run writes a recommendation artifact with per-rule suggested actions alongside its records, verified against the documented delta schema. + - _Requirements: 4.2_ + - _Boundary: Benchmark Harness_ + - _Depends: 4.1_ + +- [ ] 4.3 Introduce the loop-owned ledger and re-key calibration to rules + - Add the loop-owned overlay ledger that records per-rule status, evidence floor, threshold, and precision, applied over the ruleset at load time. + - Re-key calibration from the first-tag key to the rule identifier and write status flips into the ledger rather than only demoting ranking. + - Observable completion: an empty ledger leaves behavior byte-identical to the prior phase while a ledger entry overlays a rule's status at load, proven by an integration test. + - _Requirements: 2.4, 5.6, 6.2_ + - _Boundary: RulesetStore, Calibration subsystem_ + - _Depends: 3.1, 3.2_ + +- [ ] 4.4 Convert false-positive confirmation into a reachability knob + - When confirming a false positive, lower the finding's effective reachability multiplier rather than delete the rule, so recall stays high while the score stops over-penalizing. + - Observable completion: confirming a false positive lowers the finding's effective reachability multiplier and the rule still fires, proven by a calibration test. + - _Requirements: 5.6_ + - _Boundary: Calibration subsystem, ScoreModel_ + - _Depends: 4.3, 2.1_ + +## Phase 3 — Integration: HarnessX Execution & Self-Improvement (optional extra) + +- [ ] 5. Gate and package the HarnessX dependency-weight decision +- [ ] 5.1 Run the dependency dry-run audit and record the transitive graph + - Before adopting the HarnessX execution stages, run a dependency dry-run audit and record the realized transitive dependency graph. + - If the audited graph pulls a browser-binary, container-SDK, or web-server stack as a mandatory dependency and it is judged unacceptable, select the pre-specified vendored lean subset under the same extra name with the heavy subtrees removed. + - Observable completion: the audit produces a recorded transitive-dependency graph and a documented accept-extra-or-vendor-subset decision before any HarnessX execution code is wired in. + - _Requirements: 7.3, 7.4, 7.7_ + - _Boundary: Packaging Configuration_ + +- [ ] 5.2 Configure HarnessX as a SHA-pinned optional extra with a capability guard + - Keep the core dependency list empty and expose HarnessX only as an optional extra pinned to a specific commit revision; initialize any submodule non-recursively so the large RL fork subtree is not pulled. + - Guard every HarnessX import behind a capability check and import it lazily so heavy code paths stay cold in the default install. + - Observable completion: the default install resolves with an empty core dependency list and HarnessX symbols are only imported after the capability check passes, verified by an import-cold test. + - _Requirements: 7.1, 7.2, 7.6_ + - _Boundary: Packaging Configuration_ + - _Depends: 5.1_ + +- [ ] 6. Adopt HarnessX at the hunter loop and verifier +- [ ] 6.1 Run the hunter pool stage on a real HarnessX agent loop + - Run the hunter pool stage on a real HarnessX run loop with the control, tool, and observability processors registered in the hunter sub-harness, invoked from the existing hunter-pool stage callback. + - Confine all asynchronous execution to this single stage so the deterministic stages remain synchronous. + - Observable completion: with the extra installed, the hunter pool produces per-target trajectories from a real agent loop while all other stages stay synchronous, verified by an integration test. + - _Requirements: 1.1, 1.4_ + - _Boundary: HxScanOrchestrator_ + - _Depends: 5.2_ + +- [ ] 6.2 Map stage events and identifiers onto HarnessX hooks + - Map the existing stage-start/stage-end events to the HarnessX step-level hooks and the scan identifier to the HarnessX run identifier without losing trace continuity. + - Observable completion: a HarnessX-backed run preserves trace continuity with stage events mapped to step hooks and the scan id mapped to the run id, verified by inspecting the emitted trace. + - _Requirements: 1.7_ + - _Boundary: HxScanOrchestrator_ + - _Depends: 6.1_ + +- [ ] 6.3 (P) Run deterministic stages as model-disabled HarnessX processors with slot contracts + - Run each deterministic stage under HarnessX with model invocation disabled so no deterministic stage incurs a model call. + - Preserve each stage's declared read/write slot allow-list and validate slot access on every lifecycle hook. + - Observable completion: deterministic stages execute under HarnessX with no model calls and a slot-contract violation is raised on undeclared slot access, proven by a slot-contract test. + - _Requirements: 1.5, 1.6_ + - _Boundary: HxScanOrchestrator_ + - _Depends: 6.1_ + +- [ ] 6.4 (P) Add the LLM-judge verifier with a structural fallback + - Produce verification verdicts using the HarnessX llm-judge helpers when the extra is present, mapping the strict-JSON verdict to a verification result. + - Retain the existing structural verifier (which rejects on unknown reachability) verbatim as the zero-dependency fallback. + - Observable completion: with the extra present the judge produces verdicts and with it absent the structural verifier produces equivalent verdicts, proven by a paired verifier test. + - _Requirements: 1.2_ + - _Boundary: VerifyProcessor_ + - _Depends: 5.2_ + +- [ ] 6.5 Fall back gracefully when a HarnessX capability is unavailable + - When a HarnessX-backed stage is enabled but its capability is unavailable at runtime, fall back to the existing equivalent stage and record the degradation in the scan manifest. + - Observable completion: requesting a HarnessX stage whose capability is missing runs the existing equivalent and records a degradation entry in the manifest, proven by a fallback test. + - _Requirements: 1.8_ + - _Boundary: HxScanOrchestrator_ + - _Depends: 6.1, 6.4_ + +- [ ] 7. Stand up the bounded self-improvement loop +- [ ] 7.1 Add the two bounded edit levers + - Expose exactly two new edit levers: a rule lever covering rule status, evidence floor, and threshold, and a policy lever covering evidence floor, scope, and the score constants. + - Observable completion: the loop's valid-lever set includes exactly the rule and policy levers and rejects any edit outside their declared fields, verified by a lever test. + - _Requirements: 3.1_ + - _Boundary: EvolveValidator, improve/evolve.py_ + - _Depends: 5.2_ + +- [ ] 7.2 Enforce the rule/policy edit safety bounds in the validator + - Reject any loop edit that modifies detection pattern text and any edit that modifies the authoritative 0-5 CWE severity. + - Reject moving a rule directly from enabled to disabled, requiring shadow as a mandatory intermediate staging state. + - Assert that each edit parses, that status is one of the three literals, that threshold/evidence-floor changes stay within configured bounds, that an added rule does not duplicate an existing identifier, and that every referenced CWE resolves in policy; raise a rule-change validation error and reject the whole round on any failure. + - Observable completion: each prohibited edit (pattern change, severity change, enabled-to-disabled jump, duplicate identifier, unmapped CWE, out-of-bounds threshold) is rejected and rejects the round, proven by validator unit tests. + - _Requirements: 3.2, 3.3, 3.4, 3.5, 3.8_ + - _Boundary: EvolveValidator_ + - _Depends: 7.1, 1.1_ + +- [ ] 7.3 Add the novelty, replay, and journal gates + - Block re-proposing a previously reverted edit that lacks a retry rationale via the novelty gate. + - Run a replay smoke gate before any benchmark re-run proving the edited ruleset and policy still load and boot. + - Record each rule and policy edit, its lever, and its outcome in the journal so every change is attributable and reversible. + - Observable completion: a reverted re-proposal without rationale is blocked, a non-booting edited ruleset is rejected before re-benchmark, and every edit appears in the journal with its lever and outcome, proven by gate tests. + - _Requirements: 3.6, 3.7, 3.9_ + - _Boundary: EvolveValidator, improve/evolve.py_ + - _Depends: 7.2_ + +- [ ] 7.4 Convert benchmark deltas into loop signals + - Convert the per-rule recommendation delta and false-positive learnings into a loop signal file, writing recall-side items as miss signals and precision-side items as verifier-reject or false-positive signals. + - Observable completion: a benchmark run produces a signal file whose recall-side entries are miss signals and precision-side entries are verifier-reject/false-positive signals, verified against the documented signal schema. + - _Requirements: 4.3_ + - _Boundary: improve/evolve.py_ + - _Depends: 4.2, 7.1_ + +- [ ] 7.5 Implement the hard acceptance gate with byte-for-byte revert + - After the meta-agent edits rules or policy, re-run the benchmark and accept the round only if aggregate recall is at least 90% AND false-positive rate is under 10% AND the project score did not regress on the held set AND no unpredicted regression occurred. + - On any failed acceptance condition, revert the round and restore the rule and policy configuration byte-for-byte. + - Observable completion: a round that regresses recall is reverted and the configuration is restored byte-for-byte, while a compliant round is accepted and persisted, proven by a self-improvement integration test. + - _Requirements: 4.4, 4.5, 6.3_ + - _Boundary: improve/evolve.py_ + - _Depends: 7.3, 7.4, 4.1_ + +- [ ] 7.6 Auto-shadow precision-draggers, nominate coverage gaps, and separate the objective + - Auto-shadow a rule that drags aggregate precision below the gate rather than delete it, preserving recall while removing the false-positive cost. + - Nominate a new or loosened rule through the signal file when a coverage gap surfaces as a persistent miss, leaving pattern authoring to a human pull request. + - Treat maximizing the project score as the optimization objective and the recall/FP gate as a separate hard feasibility constraint that is never folded into the score reward. + - Observable completion: a deliberately noisy rule is auto-shadowed (not deleted) when it drags precision, a persistent miss emits a rule nomination, and the loop cannot raise the score by shadowing rules that breach the gate, proven by integration tests. + - _Requirements: 4.6, 4.7, 4.8_ + - _Boundary: improve/evolve.py_ + - _Depends: 7.5_ + +## Phase 4 — Validation: Cross-Cutting Gates & Non-Regression + +- [ ] 8. Lock the detection-gate non-regression checks into CI +- [ ] 8.1 Enforce the per-merge recall/FP detection gate + - Evaluate aggregate recall and aggregate false-positive rate on the benchmark corpus on every merge and fail the merge if recall is below 90% or the false-positive rate is at or above 10%. + - Treat the recall/FP gate as a hard constraint that is never folded into the project-score reward, blocking any change that breaches the gate regardless of score improvement. + - Observable completion: a CI run fails the merge when recall drops below 90% or false-positive rate reaches 10%, and a score improvement cannot unblock a gate-breaching change, proven by pipeline tests. + - _Requirements: 6.1, 6.5, 6.6_ + - _Boundary: CI Pipeline_ + - _Depends: 4.1_ + +- [ ] 8.2 (P) Assert byte-identical benchmark baseline for the zero-dep phases + - With the default ruleset and an empty loop ledger, assert the benchmark output is byte-identical to the pre-feature baseline for the governance and rules-as-data phases. + - Observable completion: a golden-diff test confirms byte-identical benchmark output with the default ruleset and empty ledger. + - _Requirements: 6.2_ + - _Boundary: Benchmark Harness, CI Pipeline_ + - _Depends: 3.4, 4.3_ + +- [ ] 8.3 (P) Confirm HarnessX-backed stages stay within detection tolerance + - When a HarnessX-backed stage is enabled, confirm recall and false-positive rate stay within the configured tolerance of the equivalent zero-dependency baseline. + - Observable completion: a CI run with a HarnessX-backed stage enabled keeps recall/FP within the configured tolerance of the baseline, proven by a tolerance test. + - _Requirements: 6.4_ + - _Boundary: CI Pipeline, HxScanOrchestrator_ + - _Depends: 6.1, 8.1_ + +- [ ] 8.4 (P) Add the zero-dependency guard test + - Run a guard test confirming that with HarnessX absent the governance, scoring, and benchmark planes operate and CI gating runs in the default install. + - Observable completion: the guard test passes with HarnessX uninstalled and fails if any governance/scoring/benchmark path imports HarnessX. + - _Requirements: 7.5_ + - _Boundary: CI Pipeline_ + - _Depends: 8.1_ + +- [ ] 9. Validate scan-pipeline integrity and serialization +- [ ] 9.1 (P) Verify capability fallback and unchanged sync-driver behavior + - Confirm that with the extra absent, quick and standard scans execute through the existing sync driver with behavior unchanged from the pre-adoption baseline, and that a requested-but-unavailable HarnessX stage records a degradation. + - Observable completion: an extra-absent scan matches the pre-adoption baseline and a missing-capability request records a manifest degradation, proven by an end-to-end fallback test. + - _Requirements: 1.3, 1.8_ + - _Boundary: HxScanOrchestrator, CI Pipeline_ + - _Depends: 6.5_ + +- [ ] 9.2 (P) Verify slot serialization round-trips on resume + - Confirm each scan slot type serializes and restores without loss after a resume, guarding against non-serializable slots restoring as empty. + - Observable completion: a per-slot-type round-trip test passes for every scan slot after a resume cycle. + - _Requirements: 1.5_ + - _Boundary: HxScanOrchestrator_ + - _Depends: 6.3_