feat(harnessx): Phase 3 task 6 — hunter sub-harness + llm-judge verifier + fallback

Adopt HarnessX at the hunter loop and verifier, capability-gated so the default
zero-dependency scan is byte-identical and CI (no extra) exercises the fallback.

- 6.1/6.2 build_sast() (single, lazy HarnessX composition site in harness_ext.py,
  mirrors DocuHarnessX make_docgen) + HxScanOrchestrator (hunter_harness.py): a
  drop-in for run_hunter_pool that runs one bounded HarnessX agent task per ranked
  target (ModelConfig(main=AnthropicProvider).agentic(build_sast()).run(BaseTask)),
  parses findings from a JSON structured-output contract, returns the existing
  HunterPoolResult, maps scan_id->session_id and emits per-target exit_reason/cost/
  steps to the outer trace.
- 6.4 verify_judge.verify_findings_dispatch: a drop-in superset of
  verification.verify_findings — structural verifier verbatim by default (rejects
  unknown reachability), HarnessX llm-judge (build_judge_prompt/parse_judge_response)
  when the extra + a verifier model are present; verdict mapped to VerificationResult
  evidence-gated (plausible>=0.6 accepted; unsupported/refused rejected; infra
  failure never auto-rejects).
- 6.5 cli capability gate (hx_hunter/hx_verify): selects HxScanOrchestrator /
  judge only in standard mode with a configured model AND has_harnessx(); otherwise
  falls back and records a {stage,reason:harnessx_extra_unavailable} degradation in
  the manifest (new append-only write_manifest degradations kwarg). With no model
  configured the manifest stays byte-identical.
- pyproject: mypy override ignore_missing_imports for harnessx.* (absent in the
  type-check env).

Verified the HarnessX path constructs against the real extra in a throwaway venv
(build_sast -> HarnessConfig; orchestrator -> Harness; build_judge_prompt/
parse_judge_response/verdict mapping) — only the live agent.run()/provider.complete()
LLM calls are offline-untestable and stay gated. 109 tests pass (2 construction
smokes skip without the extra), ruff + mypy clean, detection gate 93.6%/0% intact.
Tasks 6.1/6.2/6.4/6.5 done; 6.3 (deterministic stages as HarnessX processors) deferred.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-22 16:29:52 +02:00
co-authored by Claude Opus 4.8
parent 5e206df166
commit ef59c0ae3d
11 changed files with 566 additions and 11 deletions
@@ -1,7 +1,7 @@
{
"feature_name": "harnessx-self-improving-rulesets",
"created_at": "2026-06-22T10:07:18Z",
"updated_at": "2026-06-22T14:25:00Z",
"updated_at": "2026-06-22T15:05:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -139,7 +139,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
- _Depends: 5.1_
- [ ] 6. Adopt HarnessX at the hunter loop and verifier
- [ ] 6.1 Run the hunter pool stage on a real HarnessX agent loop
- [x] 6.1 Run the hunter pool stage on a real HarnessX agent loop
- Run the hunter pool stage on a real HarnessX run loop with the control, tool, and observability processors registered in the hunter sub-harness, invoked from the existing hunter-pool stage callback.
- Confine all asynchronous execution to this single stage so the deterministic stages remain synchronous.
- Observable completion: with the extra installed, the hunter pool produces per-target trajectories from a real agent loop while all other stages stay synchronous, verified by an integration test.
@@ -147,7 +147,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
- _Boundary: HxScanOrchestrator_
- _Depends: 5.2_
- [ ] 6.2 Map stage events and identifiers onto HarnessX hooks
- [x] 6.2 Map stage events and identifiers onto HarnessX hooks
- Map the existing stage-start/stage-end events to the HarnessX step-level hooks and the scan identifier to the HarnessX run identifier without losing trace continuity.
- Observable completion: a HarnessX-backed run preserves trace continuity with stage events mapped to step hooks and the scan id mapped to the run id, verified by inspecting the emitted trace.
- _Requirements: 1.7_
@@ -162,7 +162,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
- _Boundary: HxScanOrchestrator_
- _Depends: 6.1_
- [ ] 6.4 (P) Add the LLM-judge verifier with a structural fallback
- [x] 6.4 (P) Add the LLM-judge verifier with a structural fallback
- Produce verification verdicts using the HarnessX llm-judge helpers when the extra is present, mapping the strict-JSON verdict to a verification result.
- Retain the existing structural verifier (which rejects on unknown reachability) verbatim as the zero-dependency fallback.
- Observable completion: with the extra present the judge produces verdicts and with it absent the structural verifier produces equivalent verdicts, proven by a paired verifier test.
@@ -170,7 +170,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
- _Boundary: VerifyProcessor_
- _Depends: 5.2_
- [ ] 6.5 Fall back gracefully when a HarnessX capability is unavailable
- [x] 6.5 Fall back gracefully when a HarnessX capability is unavailable
- When a HarnessX-backed stage is enabled but its capability is unavailable at runtime, fall back to the existing equivalent stage and record the degradation in the scan manifest.
- Observable completion: requesting a HarnessX stage whose capability is missing runs the existing equivalent and records a degradation entry in the manifest, proven by a fallback test.
- _Requirements: 1.8_