docs(spec): Phase 3 task 5.1 — HarnessX dependency audit + DocuHarnessX blueprint

Run and record the HarnessX dependency dry-run audit and assess blueprint
alignment with the author's DocuHarnessX, to steer OpenUltraSAST's HarnessX
adoption from opencode/claude-code.

- dependency-audit.md: records the realized transitive graph (118 packages from
  30 mandatory HarnessX deps). Key finding: every heavy backend (docker/e2b/
  playwright/office-PDF) is lazily imported and the gateway is a separate dist, so
  the tree is an install-time (not runtime) cost; ~55-60 packages are dead weight
  for SAST. Decision: ACCEPT as the optional SHA-pinned extra openultrasast[harnessx]
  (core stays zero-dep; lazy heavy imports; matches DocuHarnessX precedent), with a
  fully-specified vendored lean subset as the triggered fallback and upstream
  dependency-extras as the preferred long-term slimming path. Records the
  security-tool residual risks (browser binary, docker socket, web server, CVE
  surface) and their mitigations.
- blueprint-alignment.md: DocuHarnessX <-> OpenUltraSAST assessment. They share the
  spine (one composition factory, model bound in CLI via ModelConfig.agentic,
  content-neutral processors over typed slots, project-loaded domain/"ontology"
  seam, fail-closed LLM-judge firewall, Kiro fresh-subagent steering). DocuHarnessX
  realizes "self-improving multi-agent" via the firewall + claude-code/opencode
  steering loop, NOT MetaAgent.evolve (Train is future there); OpenUltraSAST builds
  the evolve loop on the same substrate. Three principled divergences (hybrid runtime
  confining HarnessX to the hunter pool; evolve reward/feasibility split + ledger;
  zero-dep core with opt-in extra) with reconciliation, plus a concrete Phase 3
  blueprint mirroring DocuHarnessX.

Checks off spec task 5.1.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

.kiro/specs/harnessx-self-improving-rulesets/blueprint-alignment.md

@@ -0,0 +1,96 @@
+# DocuHarnessX ↔ OpenUltraSAST — Blueprint Alignment
+
+How OpenUltraSAST's HarnessX integration (Phase 3+) aligns with **DocuHarnessX**
+(`/home/mc/Source/DocuHarnessX`), the author's other HarnessX project. Both are HarnessX
+bundles steered from claude-code/opencode via Kiro spec-driven development. Goal: one shared
+architecture the steering layer drives identically.
+
+## Orientation
+
+DocuHarnessX is **agent-first** — an 8-stage pipeline (`ingest → analyze → classify → plan →
+write → review → assemble → deploy`) that *is* the agent loop. OpenUltraSAST is a
+**deterministic-pipeline + agentic-plane hybrid** — a sync `HarnessRuntime` driving 13 stages,
+with HarnessX confined to the one stage that is genuinely an agent loop (the hunter pool). They
+converge on the same load-bearing patterns and diverge only where SAST's determinism and
+supply-chain posture require it.
+
+**What "self-improving multi-agent system" means in DocuHarnessX (important):** it does **not**
+use `meta_harness` / `MetaAgent.evolve` at all (the `Train` loop is explicit future / out of
+scope). Its self-improvement is realized two ways: (1) an in-product **fail-closed LLM-judge
+quality firewall** (COBESY: fixed criteria, threshold-re-derived verdicts, unavailable → fail),
+and (2) the **claude-code/opencode Kiro steering loop** (fresh implementer → adversarial
+reviewer → debugger subagents per task). OpenUltraSAST inherits both substrates and *additionally*
+builds the `MetaAgent.evolve` loop DocuHarnessX leaves unbuilt.
+
+## Shared blueprint (the spine)
+
+| Concern | DocuHarnessX | OpenUltraSAST | Verdict |
+|---|---|---|---|
+| Composition seam | `make_docgen` in `docuharnessx/bundle.py` — the only HarnessX import site; `control \| stages_builder()`; returns model-less `HarnessConfig` | `build_sast()` in `harness/build_sast.py` (mirrors `examples/coding/build_coding`), lazy behind `_has_harnessx()` | **shared** |
+| Model binding | CLI `prepare_run()`: `model_config.agentic(make_docgen(...))`; model on `harness.model_config.main`, never in config | `HxScanOrchestrator`: `asyncio.run(model_config.agentic(cfg).run(ScanTask(...)))` | **shared** |
+| Processor base | `NoOpStage(MultiHookProcessor)`, content-neutral on `step_end`, real module-level classes for `_target_` resolution | `MultiHookProcessor(skip_model=True)` + `SlotContractMixin`, real module-level classes | **shared** |
+| Inter-stage seam | `RunContext` over append-only slot constants; frozen versioned handoff dataclasses | typed `sast.*` slots + `SlotContractMixin` (`reads_slots`/`writes_slots`) | **shared** |
+| Domain / "ontology" seam | `.docuharnessx/ontology.yaml` → `Vocabulary` (Role×Subject×Intent), loaded at run start | central **CWE policy + ruleset** (`policy/CWE_Score.tsv` + `ruleset/*.toml` + `rule_policy.json`), loaded at run start | **analogous** |
+| Acceptance firewall | COBESY all-of gate; judge `passed` untrusted/re-derived; unavailable → fail | `VerifyProcessor` (llm-judge + structural fallback, rejects unknown reachability, fail-closed) | **analogous** |
+| Steering | Kiro SDD; `CLAUDE.md`; `.kiro/{steering,specs}`; `kiro-impl` fresh-subagent + adversarial review; `kiro-verify-completion` | same Kiro convention; `.kiro/specs/harnessx-self-improving-rulesets/*`; OpenCode steers, kiro-* maintainer-only | **shared** |
+| Self-improvement (`evolve`) | **none** — future seam | `improve/evolve.py → MetaAgent.evolve`, bounded `rule`/`policy` levers, `rule_policy.json` ledger | **diverges (OUS builds it)** |
+| Dependency posture | `harnessx` direct git dep + pyyaml | core zero-dep; `harnessx` optional SHA-pinned extra, lazy | **diverges (by design)** |
+
+## Three principled divergences (and reconciliation)
+
+1. **Agent-first vs hybrid.** OUS keeps the sync `HarnessRuntime` through Phases 0–4 and confines
+   HarnessX to the hunter pool (the runtimes are "incompatible at every load-bearing point —
+   bridge, do not map"). Reconcile by treating `build_sast()` as building **only the hunter
+   sub-harness**, applying DocuHarnessX's `make_docgen` pattern *at the hunter-pool boundary*.
+   Optional Phase 5 converges OUS toward DocuHarnessX's agent-first end-state (retire
+   `HarnessRuntime`) — the divergence is temporal, and DocuHarnessX is the north star.
+2. **Editable ontology vs authoritative severity.** DocuHarnessX's whole vocabulary is editable
+   presets. OUS splits its domain seam into a **mutable detection vocabulary** (`ruleset/*.toml`,
+   loop- and human-editable) and an **immutable severity/scope authority** (`CWE_Score.tsv`,
+   verycode-upstream, the loop must never edit). Where DocuHarnessX relies on convention ("never a
+   hardcoded literal"), OUS relies on a **validator** (`EvolveValidator._policy` forbids editing the
+   0–5 severity) because a loop, not just a human, edits it.
+3. **Firewall-as-reward vs reward/feasibility split.** DocuHarnessX's firewall *is* the signal
+   (binary pass/fail). OUS must feed `MetaAgent.evolve`, so it keeps a **scalar reward**
+   (`project_score`) **separate from** a **hard feasibility constraint** (≥90% recall / <10% FP),
+   un-folded so the loop can't game the number. Reuse DocuHarnessX's fail-closed firewall verbatim
+   at the *per-finding* `VerifyProcessor` granularity; layer the reward/feasibility split on top.
+
+## Phase 3 blueprint for OpenUltraSAST (mirror DocuHarnessX)
+
+- **`build_sast()` factory** = the single HarnessX composition import site (DocuHarnessX's "blast
+  radius is this file" rule on `bundle.py`); `control | hunter_stages_builder()`; raised loop
+  thresholds (large-repo scanning revisits shapes, like large-repo docs); model bound later in
+  `HxScanOrchestrator`. OUS delta: the `_has_harnessx()` guard + raise-then-fallback to
+  `run_hunter_pool()`.
+- **Ontology seam** = CWE policy + ruleset as OUS's "ontology": `load_ruleset(path, ledger)` +
+  `load_policy(tsv)` loaded into `sast.*` slots at run start; `resolve_severity()` is the single
+  severity authority (mirrors DocuHarnessX's "every transform reads the loaded store, never a
+  literal"); absent ledger → byte-identical baseline; unmapped enabled-rule CWE → fail-loud
+  (stricter than DocuHarnessX, because severity is authoritative).
+- **Processor set**: register HarnessX classes directly **only inside the hunter sub-harness**
+  (LoopDetection, CostGuard/TokenBudget, ToolFailureGuard, ToolFilter, SelfVerify, EpisodeMetrics/
+  OTel/Checkpoint); everything deterministic is a `skip_model=True` `MultiHookProcessor` over slots
+  (SarifIngest/Preprocess/EntryPoint/Rank/Calibrate/QuickFindings/PolicyScoring/Verify/Score) with
+  stable `make_<stage>_processor()` factories so a spec replaces exactly one stub in place.
+- **`MetaAgent.evolve` loop** (Phase 4, on the same substrate): two bounded levers (`rule`,
+  `policy`); reward = `project_score`; feasibility = recall/FP gate (un-folded); signal path
+  benchmark per-rule precision/recall + `rule_policy_delta` → `rule_signals.json` →
+  `MetaAgent.evolve`; gate = validity (canonicalize → contract → `_rules`/`_policy` → replay) →
+  novelty/evidence → re-benchmark → attribution → accept or **byte-for-byte revert** (the
+  structural analog of DocuHarnessX's `DEFAULT_UNAVAILABLE_VERDICT="fail"`); ledger =
+  `rule_policy.json` (OUS's resume-safe analog of DocuHarnessX's markdown round journal).
+- **Steering surface**: reuse DocuHarnessX's `kiro-*` skills verbatim (maintainer-only); add
+  SAST-domain `.kiro/steering/` rules pinning the detection gate as a hard invariant, the zero-dep
+  guard, and the `pip --dry-run` audit gate; traceable per-module docstrings citing spec Req/Decision
+  numbers.
+
+## Net
+
+The two projects already share the spine — one composition factory, model bound in the CLI via
+`ModelConfig.agentic`, content-neutral processors over typed slots, a project-loaded domain seam,
+a fail-closed acceptance firewall, and the Kiro fresh-subagent steering loop. OpenUltraSAST diverges
+in exactly three principled places (hybrid runtime confining HarnessX to the hunter pool; it *builds*
+the `MetaAgent.evolve` loop with a reward/feasibility split + `rule_policy.json` ledger; zero-dep core
+with HarnessX opt-in). Adopting DocuHarnessX's load-bearing patterns at the points above gives the
+two projects one architecture that opencode/claude-code steers identically.

.kiro/specs/harnessx-self-improving-rulesets/dependency-audit.md

@@ -0,0 +1,116 @@
+# HarnessX Dependency Dry-Run Audit (Task 5.1)
+
+Recorded realized transitive dependency graph and the accept-vs-vendor decision for
+adopting HarnessX (`git+https://github.com/Darwin-Agent/HarnessX.git`) in Phase 3.
+
+## Method
+
+```bash
+uv venv /tmp/hx-audit --python 3.11
+uv pip install --dry-run --python /tmp/hx-audit/bin/python <HarnessX checkout>
+```
+
+- **118 transitive packages** resolved from HarnessX's **30 direct** `[project.dependencies]`
+  (all mandatory upstream; the only extra is `dev`).
+- Pinned versions observed: `harnessx 0.1.0`, and engine image `registry.dagger.io`/SDK
+  unrelated; LLM SDKs `anthropic 0.111.0`, `openai 2.43.0`, `litellm 1.89.3`.
+
+### Realized transitive graph (118)
+
+```
+aiofiles aiohappyeyeballs aiohttp aiosignal annotated-doc annotated-types anthropic
+antlr4-python3-runtime anyio attrs bracex certifi cffi charset-normalizer click cryptography
+distro docker dockerfile-parse docstring-parser e2b et-xmlfile fastapi fastuuid filelock
+frozenlist fsspec googleapis-common-protos greenlet grpcio h11 h2 harnessx hf-xet hpack
+html2text httpcore httptools httpx httpx-sse huggingface-hub hydra-core hyperframe idna
+importlib-metadata jinja2 jiter jsonschema jsonschema-specifications litellm loguru lxml
+markdown-it-py markupsafe mcp mdurl multidict omegaconf openai openpyxl opentelemetry-api
+opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-common
+opentelemetry-exporter-otlp-proto-grpc opentelemetry-exporter-otlp-proto-http opentelemetry-proto
+opentelemetry-sdk opentelemetry-semantic-conventions packaging pdfminer-six pdfplumber pillow
+playwright prompt-toolkit propcache protobuf pycparser pydantic pydantic-core pydantic-settings
+pyee pygments pyjwt pypdfium2 python-dateutil python-docx python-dotenv python-multipart
+python-pptx pyyaml referencing regex reportlab requests rich rpds-py shellingham six sniffio
+sse-starlette starlette structlog tiktoken tokenizers tqdm typer typing-extensions
+typing-inspection urllib3 uvicorn uvloop watchfiles wcmatch wcwidth websockets xlsxwriter yarl zipp
+```
+
+## Key finding: heavy backends are lazy; ~half the tree is dead weight
+
+Every heavy backend is **lazily imported** — `docker`/`e2b` are `TYPE_CHECKING`-guarded and
+imported inside provider factories; `playwright` is imported inside `_get_local_page()`;
+office/PDF libs inside `tools/builtin/read.py` parse functions; `sandbox/__init__.py` registers
+docker/e2b via `try/except`. The `gateway/` is a **separate distribution**. So the 118-package
+tree is an **install-time** cost; at **runtime only what a given agent path touches is loaded**.
+
+This is the answer to "I don't need most of these": neither OpenUltraSAST (SAST) nor the
+author's DocuHarnessX (docs) loads the bulk of it — HarnessX simply declares every capability as
+a *mandatory* dependency.
+
+| Category | Packages | SAST verdict |
+|---|---|---|
+| LLM SDKs | anthropic, openai, litellm, tiktoken, tokenizers, huggingface-hub, hf-xet, jiter, regex, tqdm, fsspec | **needed** |
+| HTTP / async / pydantic core | httpx(+core/h11/h2/hpack/hyperframe), anyio, aiohttp(+glue), pydantic(+core/settings), certifi, urllib3, requests, sniffio | **needed** |
+| Config / CLI / logging | hydra-core, omegaconf, antlr4-runtime, typer, click, rich, prompt-toolkit, structlog, loguru, jsonschema(+glue) | **needed** |
+| Browser automation | **playwright, pyee, greenlet, html2text** | **dead weight** (+ out-of-band Chromium binary download via `playwright install`) |
+| Sandbox backends | docker, dockerfile-parse, **e2b** | docker = useful (isolate hunter); **e2b = dead weight** |
+| Web gateway / Lab UI | **fastapi, uvicorn, starlette, sse-starlette, httptools, websockets, uvloop, watchfiles, python-multipart, aiofiles** | **dead weight** (server surface; never started by a headless embed) |
+| gRPC + OpenTelemetry | grpcio, protobuf, googleapis-common-protos, opentelemetry-* (8) | **optional** (tracing only) |
+| Office / PDF | **python-pptx, python-docx, openpyxl, reportlab, pdfplumber, pdfminer-six, pypdfium2, pillow, xlsxwriter, lxml, et-xmlfile** | **dead weight** (SAST reads source, not documents) |
+| MCP | mcp, httpx-sse | optional (only if the agent calls MCP tools) |
+
+Roughly **55–60 packages are opt-in capability weight** pulled in mandatorily, including a
+browser binary and a Docker SDK — an awkward footprint for a supply-chain auditing tool.
+
+## Decision
+
+**ACCEPT HarnessX as the optional, SHA-pinned extra `openultrasast[harnessx]`, lazy-imported
+behind `_has_harnessx()`. Keep a pre-specified vendored lean subset as a documented, triggered
+fallback. Pursue upstream dependency-extras as the preferred long-term slimming path.**
+
+Rationale:
+1. **Footprint is fully opt-in.** Core `dependencies = []` stays zero-dep; the governance /
+   scoring / benchmark / detection planes never pull HarnessX. Only `openultrasast[harnessx]`
+   installs the tree, and lazy imports keep heavy backends cold unless the path runs.
+2. **Author precedent.** DocuHarnessX already ships `harnessx` as a direct git dependency — the
+   `ModelConfig.agentic` integration is proven and maintained. OpenUltraSAST reuses the
+   mechanism but tightens the posture (opt-in, not direct).
+3. **Lower maintenance than vendoring.** Accept + SHA-pin gives reproducibility without a fork;
+   vendoring is paid only if the *installed* graph is itself judged unacceptable.
+
+### Preferred long-term: upstream dependency-extras (helps both projects)
+
+The clean fix for "I don't need most of these" is to modularize HarnessX's own dependencies into
+extras — e.g. `harnessx[browser]`, `harnessx[sandbox-e2b]`, `harnessx[gateway]`, `harnessx[office]`,
+`harnessx[otel]` — leaving a lean agent-runtime core (`anthropic`/`openai`/`litellm`/`pydantic`/
+`httpx`/`hydra`/`structlog`/`mcp`). Since the author owns the `Darwin-Agent/HarnessX` line, this is
+actionable and benefits DocuHarnessX equally. OpenUltraSAST would then depend on
+`harnessx` (lean core) under its `[harnessx]` extra and add only the subsets it needs.
+
+### Vendored lean subset (fallback, fully specified)
+
+If triggered, vendor `harnessx/` only:
+
+- **KEEP**: `core/`, `processors/{control,evaluation,observability,tools,context,memory,multi_model}`,
+  `meta_harness/`, `rl/`, `providers/`, `tools/builtin/` (minus `browser.py`/`web_*`),
+  `sandbox/{base,local,docker}.py`, plus `config`/`tracing`/`plugins`/`bundles`/`workspace` as needed.
+- **REMOVE**: `frontend/`, `gateway/`, `harnessx/api/` (drops the web-server stack), `examples/`,
+  `docs/`, `benchmarks/`, `container/`, `scripts/`, `recipe/` (incl. the `recipe/verl_harnessX/verl`
+  submodule — clone non-recursive regardless), `extensions/skills/{docx,pdf,pptx,xlsx}` (drops
+  office/PDF), `harnessx/sandbox/e2b.py` (drops e2b), `harnessx/tools/builtin/browser.py`
+  (drops playwright + browser binary). Net ≈ 40–60% fewer packages, no browser binary.
+
+## Residual risks (security-tool-specific) and mitigations
+
+| Risk | Mitigation |
+|---|---|
+| `playwright install` fetches a Chromium binary out-of-band (unaudited, breaks hermetic builds) | Browser path is lazy and never invoked; document that `[harnessx]` does not trigger a browser download; lean subset deletes `browser.py` |
+| `docker` SDK can reach the Docker daemon socket (privilege boundary) | Docker sandbox **default-off**, `LocalSandbox` default; run untrusted target code only inside the sandbox; SHA-pin docker/cryptography/cffi |
+| FastAPI/uvicorn listening-socket surface in a security tool | `api/` server only starts on the explicit `serve` path; add a guard test asserting no server starts on import; lean subset removes `api/` |
+| 118-package CVE surface in a supply-chain tool | SHA-pin the entire extra; keep core **zero-dep**; add a **zero-dep CI guard test** (importing `openultrasast` without the extra pulls nothing outside stdlib) |
+| `recipe/verl` git submodule recursion | Init non-recursively; never in the accept-as-extra install path |
+
+> Decision date 2026-06-22. The `pip install --dry-run` audit is itself the Phase-3 gate: it is
+> re-run before the packaging task (5.2), and if the realized graph pulls a browser binary /
+> Docker SDK / FastAPI stack as *mandatory* (non-lazy) deps judged unacceptable, the spec switches
+> to the vendored lean subset under the same extra name.

.kiro/specs/harnessx-self-improving-rulesets/spec.json

@@ -1,7 +1,7 @@
 {
   "feature_name": "harnessx-self-improving-rulesets",
   "created_at": "2026-06-22T10:07:18Z",
-  "updated_at": "2026-06-22T13:45:00Z",
+  "updated_at": "2026-06-22T14:05:00Z",
   "language": "en",
   "phase": "implementation",
   "approvals": {

.kiro/specs/harnessx-self-improving-rulesets/tasks.md

@@ -123,7 +123,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
 ## Phase 3 — Integration: HarnessX Execution & Self-Improvement (optional extra)
 
 - [ ] 5. Gate and package the HarnessX dependency-weight decision
-- [ ] 5.1 Run the dependency dry-run audit and record the transitive graph
+- [x] 5.1 Run the dependency dry-run audit and record the transitive graph
   - Before adopting the HarnessX execution stages, run a dependency dry-run audit and record the realized transitive dependency graph.
   - If the audited graph pulls a browser-binary, container-SDK, or web-server stack as a mandatory dependency and it is judged unacceptable, select the pre-specified vendored lean subset under the same extra name with the heavy subtrees removed.
   - Observable completion: the audit produces a recorded transitive-dependency graph and a documented accept-extra-or-vendor-subset decision before any HarnessX execution code is wired in.