From 6aa5a1bc962ae680adb3aee6b42afa1c0773ca77 Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Mon, 22 Jun 2026 15:58:30 +0200 Subject: [PATCH] =?UTF-8?q?docs(spec):=20Phase=203=20task=205.1=20?= =?UTF-8?q?=E2=80=94=20HarnessX=20dependency=20audit=20+=20DocuHarnessX=20?= =?UTF-8?q?blueprint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run and record the HarnessX dependency dry-run audit and assess blueprint alignment with the author's DocuHarnessX, to steer OpenUltraSAST's HarnessX adoption from opencode/claude-code. - dependency-audit.md: records the realized transitive graph (118 packages from 30 mandatory HarnessX deps). Key finding: every heavy backend (docker/e2b/ playwright/office-PDF) is lazily imported and the gateway is a separate dist, so the tree is an install-time (not runtime) cost; ~55-60 packages are dead weight for SAST. Decision: ACCEPT as the optional SHA-pinned extra openultrasast[harnessx] (core stays zero-dep; lazy heavy imports; matches DocuHarnessX precedent), with a fully-specified vendored lean subset as the triggered fallback and upstream dependency-extras as the preferred long-term slimming path. Records the security-tool residual risks (browser binary, docker socket, web server, CVE surface) and their mitigations. - blueprint-alignment.md: DocuHarnessX <-> OpenUltraSAST assessment. They share the spine (one composition factory, model bound in CLI via ModelConfig.agentic, content-neutral processors over typed slots, project-loaded domain/"ontology" seam, fail-closed LLM-judge firewall, Kiro fresh-subagent steering). DocuHarnessX realizes "self-improving multi-agent" via the firewall + claude-code/opencode steering loop, NOT MetaAgent.evolve (Train is future there); OpenUltraSAST builds the evolve loop on the same substrate. Three principled divergences (hybrid runtime confining HarnessX to the hunter pool; evolve reward/feasibility split + ledger; zero-dep core with opt-in extra) with reconciliation, plus a concrete Phase 3 blueprint mirroring DocuHarnessX. Checks off spec task 5.1. Co-Authored-By: Claude Opus 4.8 --- .../blueprint-alignment.md | 96 +++++++++++++++ .../dependency-audit.md | 116 ++++++++++++++++++ .../spec.json | 2 +- .../harnessx-self-improving-rulesets/tasks.md | 2 +- 4 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/blueprint-alignment.md create mode 100644 .kiro/specs/harnessx-self-improving-rulesets/dependency-audit.md diff --git a/.kiro/specs/harnessx-self-improving-rulesets/blueprint-alignment.md b/.kiro/specs/harnessx-self-improving-rulesets/blueprint-alignment.md new file mode 100644 index 0000000..58ca378 --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/blueprint-alignment.md @@ -0,0 +1,96 @@ +# DocuHarnessX ↔ OpenUltraSAST — Blueprint Alignment + +How OpenUltraSAST's HarnessX integration (Phase 3+) aligns with **DocuHarnessX** +(`/home/mc/Source/DocuHarnessX`), the author's other HarnessX project. Both are HarnessX +bundles steered from claude-code/opencode via Kiro spec-driven development. Goal: one shared +architecture the steering layer drives identically. + +## Orientation + +DocuHarnessX is **agent-first** — an 8-stage pipeline (`ingest → analyze → classify → plan → +write → review → assemble → deploy`) that *is* the agent loop. OpenUltraSAST is a +**deterministic-pipeline + agentic-plane hybrid** — a sync `HarnessRuntime` driving 13 stages, +with HarnessX confined to the one stage that is genuinely an agent loop (the hunter pool). They +converge on the same load-bearing patterns and diverge only where SAST's determinism and +supply-chain posture require it. + +**What "self-improving multi-agent system" means in DocuHarnessX (important):** it does **not** +use `meta_harness` / `MetaAgent.evolve` at all (the `Train` loop is explicit future / out of +scope). Its self-improvement is realized two ways: (1) an in-product **fail-closed LLM-judge +quality firewall** (COBESY: fixed criteria, threshold-re-derived verdicts, unavailable → fail), +and (2) the **claude-code/opencode Kiro steering loop** (fresh implementer → adversarial +reviewer → debugger subagents per task). OpenUltraSAST inherits both substrates and *additionally* +builds the `MetaAgent.evolve` loop DocuHarnessX leaves unbuilt. + +## Shared blueprint (the spine) + +| Concern | DocuHarnessX | OpenUltraSAST | Verdict | +|---|---|---|---| +| Composition seam | `make_docgen` in `docuharnessx/bundle.py` — the only HarnessX import site; `control \| stages_builder()`; returns model-less `HarnessConfig` | `build_sast()` in `harness/build_sast.py` (mirrors `examples/coding/build_coding`), lazy behind `_has_harnessx()` | **shared** | +| Model binding | CLI `prepare_run()`: `model_config.agentic(make_docgen(...))`; model on `harness.model_config.main`, never in config | `HxScanOrchestrator`: `asyncio.run(model_config.agentic(cfg).run(ScanTask(...)))` | **shared** | +| Processor base | `NoOpStage(MultiHookProcessor)`, content-neutral on `step_end`, real module-level classes for `_target_` resolution | `MultiHookProcessor(skip_model=True)` + `SlotContractMixin`, real module-level classes | **shared** | +| Inter-stage seam | `RunContext` over append-only slot constants; frozen versioned handoff dataclasses | typed `sast.*` slots + `SlotContractMixin` (`reads_slots`/`writes_slots`) | **shared** | +| Domain / "ontology" seam | `.docuharnessx/ontology.yaml` → `Vocabulary` (Role×Subject×Intent), loaded at run start | central **CWE policy + ruleset** (`policy/CWE_Score.tsv` + `ruleset/*.toml` + `rule_policy.json`), loaded at run start | **analogous** | +| Acceptance firewall | COBESY all-of gate; judge `passed` untrusted/re-derived; unavailable → fail | `VerifyProcessor` (llm-judge + structural fallback, rejects unknown reachability, fail-closed) | **analogous** | +| Steering | Kiro SDD; `CLAUDE.md`; `.kiro/{steering,specs}`; `kiro-impl` fresh-subagent + adversarial review; `kiro-verify-completion` | same Kiro convention; `.kiro/specs/harnessx-self-improving-rulesets/*`; OpenCode steers, kiro-* maintainer-only | **shared** | +| Self-improvement (`evolve`) | **none** — future seam | `improve/evolve.py → MetaAgent.evolve`, bounded `rule`/`policy` levers, `rule_policy.json` ledger | **diverges (OUS builds it)** | +| Dependency posture | `harnessx` direct git dep + pyyaml | core zero-dep; `harnessx` optional SHA-pinned extra, lazy | **diverges (by design)** | + +## Three principled divergences (and reconciliation) + +1. **Agent-first vs hybrid.** OUS keeps the sync `HarnessRuntime` through Phases 0–4 and confines + HarnessX to the hunter pool (the runtimes are "incompatible at every load-bearing point — + bridge, do not map"). Reconcile by treating `build_sast()` as building **only the hunter + sub-harness**, applying DocuHarnessX's `make_docgen` pattern *at the hunter-pool boundary*. + Optional Phase 5 converges OUS toward DocuHarnessX's agent-first end-state (retire + `HarnessRuntime`) — the divergence is temporal, and DocuHarnessX is the north star. +2. **Editable ontology vs authoritative severity.** DocuHarnessX's whole vocabulary is editable + presets. OUS splits its domain seam into a **mutable detection vocabulary** (`ruleset/*.toml`, + loop- and human-editable) and an **immutable severity/scope authority** (`CWE_Score.tsv`, + verycode-upstream, the loop must never edit). Where DocuHarnessX relies on convention ("never a + hardcoded literal"), OUS relies on a **validator** (`EvolveValidator._policy` forbids editing the + 0–5 severity) because a loop, not just a human, edits it. +3. **Firewall-as-reward vs reward/feasibility split.** DocuHarnessX's firewall *is* the signal + (binary pass/fail). OUS must feed `MetaAgent.evolve`, so it keeps a **scalar reward** + (`project_score`) **separate from** a **hard feasibility constraint** (≥90% recall / <10% FP), + un-folded so the loop can't game the number. Reuse DocuHarnessX's fail-closed firewall verbatim + at the *per-finding* `VerifyProcessor` granularity; layer the reward/feasibility split on top. + +## Phase 3 blueprint for OpenUltraSAST (mirror DocuHarnessX) + +- **`build_sast()` factory** = the single HarnessX composition import site (DocuHarnessX's "blast + radius is this file" rule on `bundle.py`); `control | hunter_stages_builder()`; raised loop + thresholds (large-repo scanning revisits shapes, like large-repo docs); model bound later in + `HxScanOrchestrator`. OUS delta: the `_has_harnessx()` guard + raise-then-fallback to + `run_hunter_pool()`. +- **Ontology seam** = CWE policy + ruleset as OUS's "ontology": `load_ruleset(path, ledger)` + + `load_policy(tsv)` loaded into `sast.*` slots at run start; `resolve_severity()` is the single + severity authority (mirrors DocuHarnessX's "every transform reads the loaded store, never a + literal"); absent ledger → byte-identical baseline; unmapped enabled-rule CWE → fail-loud + (stricter than DocuHarnessX, because severity is authoritative). +- **Processor set**: register HarnessX classes directly **only inside the hunter sub-harness** + (LoopDetection, CostGuard/TokenBudget, ToolFailureGuard, ToolFilter, SelfVerify, EpisodeMetrics/ + OTel/Checkpoint); everything deterministic is a `skip_model=True` `MultiHookProcessor` over slots + (SarifIngest/Preprocess/EntryPoint/Rank/Calibrate/QuickFindings/PolicyScoring/Verify/Score) with + stable `make__processor()` factories so a spec replaces exactly one stub in place. +- **`MetaAgent.evolve` loop** (Phase 4, on the same substrate): two bounded levers (`rule`, + `policy`); reward = `project_score`; feasibility = recall/FP gate (un-folded); signal path + benchmark per-rule precision/recall + `rule_policy_delta` → `rule_signals.json` → + `MetaAgent.evolve`; gate = validity (canonicalize → contract → `_rules`/`_policy` → replay) → + novelty/evidence → re-benchmark → attribution → accept or **byte-for-byte revert** (the + structural analog of DocuHarnessX's `DEFAULT_UNAVAILABLE_VERDICT="fail"`); ledger = + `rule_policy.json` (OUS's resume-safe analog of DocuHarnessX's markdown round journal). +- **Steering surface**: reuse DocuHarnessX's `kiro-*` skills verbatim (maintainer-only); add + SAST-domain `.kiro/steering/` rules pinning the detection gate as a hard invariant, the zero-dep + guard, and the `pip --dry-run` audit gate; traceable per-module docstrings citing spec Req/Decision + numbers. + +## Net + +The two projects already share the spine — one composition factory, model bound in the CLI via +`ModelConfig.agentic`, content-neutral processors over typed slots, a project-loaded domain seam, +a fail-closed acceptance firewall, and the Kiro fresh-subagent steering loop. OpenUltraSAST diverges +in exactly three principled places (hybrid runtime confining HarnessX to the hunter pool; it *builds* +the `MetaAgent.evolve` loop with a reward/feasibility split + `rule_policy.json` ledger; zero-dep core +with HarnessX opt-in). Adopting DocuHarnessX's load-bearing patterns at the points above gives the +two projects one architecture that opencode/claude-code steers identically. diff --git a/.kiro/specs/harnessx-self-improving-rulesets/dependency-audit.md b/.kiro/specs/harnessx-self-improving-rulesets/dependency-audit.md new file mode 100644 index 0000000..917e2cb --- /dev/null +++ b/.kiro/specs/harnessx-self-improving-rulesets/dependency-audit.md @@ -0,0 +1,116 @@ +# HarnessX Dependency Dry-Run Audit (Task 5.1) + +Recorded realized transitive dependency graph and the accept-vs-vendor decision for +adopting HarnessX (`git+https://github.com/Darwin-Agent/HarnessX.git`) in Phase 3. + +## Method + +```bash +uv venv /tmp/hx-audit --python 3.11 +uv pip install --dry-run --python /tmp/hx-audit/bin/python +``` + +- **118 transitive packages** resolved from HarnessX's **30 direct** `[project.dependencies]` + (all mandatory upstream; the only extra is `dev`). +- Pinned versions observed: `harnessx 0.1.0`, and engine image `registry.dagger.io`/SDK + unrelated; LLM SDKs `anthropic 0.111.0`, `openai 2.43.0`, `litellm 1.89.3`. + +### Realized transitive graph (118) + +``` +aiofiles aiohappyeyeballs aiohttp aiosignal annotated-doc annotated-types anthropic +antlr4-python3-runtime anyio attrs bracex certifi cffi charset-normalizer click cryptography +distro docker dockerfile-parse docstring-parser e2b et-xmlfile fastapi fastuuid filelock +frozenlist fsspec googleapis-common-protos greenlet grpcio h11 h2 harnessx hf-xet hpack +html2text httpcore httptools httpx httpx-sse huggingface-hub hydra-core hyperframe idna +importlib-metadata jinja2 jiter jsonschema jsonschema-specifications litellm loguru lxml +markdown-it-py markupsafe mcp mdurl multidict omegaconf openai openpyxl opentelemetry-api +opentelemetry-exporter-otlp opentelemetry-exporter-otlp-proto-common +opentelemetry-exporter-otlp-proto-grpc opentelemetry-exporter-otlp-proto-http opentelemetry-proto +opentelemetry-sdk opentelemetry-semantic-conventions packaging pdfminer-six pdfplumber pillow +playwright prompt-toolkit propcache protobuf pycparser pydantic pydantic-core pydantic-settings +pyee pygments pyjwt pypdfium2 python-dateutil python-docx python-dotenv python-multipart +python-pptx pyyaml referencing regex reportlab requests rich rpds-py shellingham six sniffio +sse-starlette starlette structlog tiktoken tokenizers tqdm typer typing-extensions +typing-inspection urllib3 uvicorn uvloop watchfiles wcmatch wcwidth websockets xlsxwriter yarl zipp +``` + +## Key finding: heavy backends are lazy; ~half the tree is dead weight + +Every heavy backend is **lazily imported** — `docker`/`e2b` are `TYPE_CHECKING`-guarded and +imported inside provider factories; `playwright` is imported inside `_get_local_page()`; +office/PDF libs inside `tools/builtin/read.py` parse functions; `sandbox/__init__.py` registers +docker/e2b via `try/except`. The `gateway/` is a **separate distribution**. So the 118-package +tree is an **install-time** cost; at **runtime only what a given agent path touches is loaded**. + +This is the answer to "I don't need most of these": neither OpenUltraSAST (SAST) nor the +author's DocuHarnessX (docs) loads the bulk of it — HarnessX simply declares every capability as +a *mandatory* dependency. + +| Category | Packages | SAST verdict | +|---|---|---| +| LLM SDKs | anthropic, openai, litellm, tiktoken, tokenizers, huggingface-hub, hf-xet, jiter, regex, tqdm, fsspec | **needed** | +| HTTP / async / pydantic core | httpx(+core/h11/h2/hpack/hyperframe), anyio, aiohttp(+glue), pydantic(+core/settings), certifi, urllib3, requests, sniffio | **needed** | +| Config / CLI / logging | hydra-core, omegaconf, antlr4-runtime, typer, click, rich, prompt-toolkit, structlog, loguru, jsonschema(+glue) | **needed** | +| Browser automation | **playwright, pyee, greenlet, html2text** | **dead weight** (+ out-of-band Chromium binary download via `playwright install`) | +| Sandbox backends | docker, dockerfile-parse, **e2b** | docker = useful (isolate hunter); **e2b = dead weight** | +| Web gateway / Lab UI | **fastapi, uvicorn, starlette, sse-starlette, httptools, websockets, uvloop, watchfiles, python-multipart, aiofiles** | **dead weight** (server surface; never started by a headless embed) | +| gRPC + OpenTelemetry | grpcio, protobuf, googleapis-common-protos, opentelemetry-* (8) | **optional** (tracing only) | +| Office / PDF | **python-pptx, python-docx, openpyxl, reportlab, pdfplumber, pdfminer-six, pypdfium2, pillow, xlsxwriter, lxml, et-xmlfile** | **dead weight** (SAST reads source, not documents) | +| MCP | mcp, httpx-sse | optional (only if the agent calls MCP tools) | + +Roughly **55–60 packages are opt-in capability weight** pulled in mandatorily, including a +browser binary and a Docker SDK — an awkward footprint for a supply-chain auditing tool. + +## Decision + +**ACCEPT HarnessX as the optional, SHA-pinned extra `openultrasast[harnessx]`, lazy-imported +behind `_has_harnessx()`. Keep a pre-specified vendored lean subset as a documented, triggered +fallback. Pursue upstream dependency-extras as the preferred long-term slimming path.** + +Rationale: +1. **Footprint is fully opt-in.** Core `dependencies = []` stays zero-dep; the governance / + scoring / benchmark / detection planes never pull HarnessX. Only `openultrasast[harnessx]` + installs the tree, and lazy imports keep heavy backends cold unless the path runs. +2. **Author precedent.** DocuHarnessX already ships `harnessx` as a direct git dependency — the + `ModelConfig.agentic` integration is proven and maintained. OpenUltraSAST reuses the + mechanism but tightens the posture (opt-in, not direct). +3. **Lower maintenance than vendoring.** Accept + SHA-pin gives reproducibility without a fork; + vendoring is paid only if the *installed* graph is itself judged unacceptable. + +### Preferred long-term: upstream dependency-extras (helps both projects) + +The clean fix for "I don't need most of these" is to modularize HarnessX's own dependencies into +extras — e.g. `harnessx[browser]`, `harnessx[sandbox-e2b]`, `harnessx[gateway]`, `harnessx[office]`, +`harnessx[otel]` — leaving a lean agent-runtime core (`anthropic`/`openai`/`litellm`/`pydantic`/ +`httpx`/`hydra`/`structlog`/`mcp`). Since the author owns the `Darwin-Agent/HarnessX` line, this is +actionable and benefits DocuHarnessX equally. OpenUltraSAST would then depend on +`harnessx` (lean core) under its `[harnessx]` extra and add only the subsets it needs. + +### Vendored lean subset (fallback, fully specified) + +If triggered, vendor `harnessx/` only: + +- **KEEP**: `core/`, `processors/{control,evaluation,observability,tools,context,memory,multi_model}`, + `meta_harness/`, `rl/`, `providers/`, `tools/builtin/` (minus `browser.py`/`web_*`), + `sandbox/{base,local,docker}.py`, plus `config`/`tracing`/`plugins`/`bundles`/`workspace` as needed. +- **REMOVE**: `frontend/`, `gateway/`, `harnessx/api/` (drops the web-server stack), `examples/`, + `docs/`, `benchmarks/`, `container/`, `scripts/`, `recipe/` (incl. the `recipe/verl_harnessX/verl` + submodule — clone non-recursive regardless), `extensions/skills/{docx,pdf,pptx,xlsx}` (drops + office/PDF), `harnessx/sandbox/e2b.py` (drops e2b), `harnessx/tools/builtin/browser.py` + (drops playwright + browser binary). Net ≈ 40–60% fewer packages, no browser binary. + +## Residual risks (security-tool-specific) and mitigations + +| Risk | Mitigation | +|---|---| +| `playwright install` fetches a Chromium binary out-of-band (unaudited, breaks hermetic builds) | Browser path is lazy and never invoked; document that `[harnessx]` does not trigger a browser download; lean subset deletes `browser.py` | +| `docker` SDK can reach the Docker daemon socket (privilege boundary) | Docker sandbox **default-off**, `LocalSandbox` default; run untrusted target code only inside the sandbox; SHA-pin docker/cryptography/cffi | +| FastAPI/uvicorn listening-socket surface in a security tool | `api/` server only starts on the explicit `serve` path; add a guard test asserting no server starts on import; lean subset removes `api/` | +| 118-package CVE surface in a supply-chain tool | SHA-pin the entire extra; keep core **zero-dep**; add a **zero-dep CI guard test** (importing `openultrasast` without the extra pulls nothing outside stdlib) | +| `recipe/verl` git submodule recursion | Init non-recursively; never in the accept-as-extra install path | + +> Decision date 2026-06-22. The `pip install --dry-run` audit is itself the Phase-3 gate: it is +> re-run before the packaging task (5.2), and if the realized graph pulls a browser binary / +> Docker SDK / FastAPI stack as *mandatory* (non-lazy) deps judged unacceptable, the spec switches +> to the vendored lean subset under the same extra name. diff --git a/.kiro/specs/harnessx-self-improving-rulesets/spec.json b/.kiro/specs/harnessx-self-improving-rulesets/spec.json index 19924bb..565dd3c 100644 --- a/.kiro/specs/harnessx-self-improving-rulesets/spec.json +++ b/.kiro/specs/harnessx-self-improving-rulesets/spec.json @@ -1,7 +1,7 @@ { "feature_name": "harnessx-self-improving-rulesets", "created_at": "2026-06-22T10:07:18Z", - "updated_at": "2026-06-22T13:45:00Z", + "updated_at": "2026-06-22T14:05:00Z", "language": "en", "phase": "implementation", "approvals": { diff --git a/.kiro/specs/harnessx-self-improving-rulesets/tasks.md b/.kiro/specs/harnessx-self-improving-rulesets/tasks.md index cbc2a62..f8ff6be 100644 --- a/.kiro/specs/harnessx-self-improving-rulesets/tasks.md +++ b/.kiro/specs/harnessx-self-improving-rulesets/tasks.md @@ -123,7 +123,7 @@ This plan splits the fused regex layer into three governed owners (rules detect, ## Phase 3 — Integration: HarnessX Execution & Self-Improvement (optional extra) - [ ] 5. Gate and package the HarnessX dependency-weight decision -- [ ] 5.1 Run the dependency dry-run audit and record the transitive graph +- [x] 5.1 Run the dependency dry-run audit and record the transitive graph - Before adopting the HarnessX execution stages, run a dependency dry-run audit and record the realized transitive dependency graph. - If the audited graph pulls a browser-binary, container-SDK, or web-server stack as a mandatory dependency and it is judged unacceptable, select the pre-specified vendored lean subset under the same extra name with the heavy subtrees removed. - Observable completion: the audit produces a recorded transitive-dependency graph and a documented accept-extra-or-vendor-subset decision before any HarnessX execution code is wired in.