mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
feat(harnessx): Phase 3 task 6 — hunter sub-harness + llm-judge verifier + fallback
Adopt HarnessX at the hunter loop and verifier, capability-gated so the default
zero-dependency scan is byte-identical and CI (no extra) exercises the fallback.
- 6.1/6.2 build_sast() (single, lazy HarnessX composition site in harness_ext.py,
mirrors DocuHarnessX make_docgen) + HxScanOrchestrator (hunter_harness.py): a
drop-in for run_hunter_pool that runs one bounded HarnessX agent task per ranked
target (ModelConfig(main=AnthropicProvider).agentic(build_sast()).run(BaseTask)),
parses findings from a JSON structured-output contract, returns the existing
HunterPoolResult, maps scan_id->session_id and emits per-target exit_reason/cost/
steps to the outer trace.
- 6.4 verify_judge.verify_findings_dispatch: a drop-in superset of
verification.verify_findings — structural verifier verbatim by default (rejects
unknown reachability), HarnessX llm-judge (build_judge_prompt/parse_judge_response)
when the extra + a verifier model are present; verdict mapped to VerificationResult
evidence-gated (plausible>=0.6 accepted; unsupported/refused rejected; infra
failure never auto-rejects).
- 6.5 cli capability gate (hx_hunter/hx_verify): selects HxScanOrchestrator /
judge only in standard mode with a configured model AND has_harnessx(); otherwise
falls back and records a {stage,reason:harnessx_extra_unavailable} degradation in
the manifest (new append-only write_manifest degradations kwarg). With no model
configured the manifest stays byte-identical.
- pyproject: mypy override ignore_missing_imports for harnessx.* (absent in the
type-check env).
Verified the HarnessX path constructs against the real extra in a throwaway venv
(build_sast -> HarnessConfig; orchestrator -> Harness; build_judge_prompt/
parse_judge_response/verdict mapping) — only the live agent.run()/provider.complete()
LLM calls are offline-untestable and stay gated. 109 tests pass (2 construction
smokes skip without the extra), ruff + mypy clean, detection gate 93.6%/0% intact.
Tasks 6.1/6.2/6.4/6.5 done; 6.3 (deterministic stages as HarnessX processors) deferred.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
5e206df166
commit
ef59c0ae3d
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "harnessx-self-improving-rulesets",
|
||||
"created_at": "2026-06-22T10:07:18Z",
|
||||
"updated_at": "2026-06-22T14:25:00Z",
|
||||
"updated_at": "2026-06-22T15:05:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -139,7 +139,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Depends: 5.1_
|
||||
|
||||
- [ ] 6. Adopt HarnessX at the hunter loop and verifier
|
||||
- [ ] 6.1 Run the hunter pool stage on a real HarnessX agent loop
|
||||
- [x] 6.1 Run the hunter pool stage on a real HarnessX agent loop
|
||||
- Run the hunter pool stage on a real HarnessX run loop with the control, tool, and observability processors registered in the hunter sub-harness, invoked from the existing hunter-pool stage callback.
|
||||
- Confine all asynchronous execution to this single stage so the deterministic stages remain synchronous.
|
||||
- Observable completion: with the extra installed, the hunter pool produces per-target trajectories from a real agent loop while all other stages stay synchronous, verified by an integration test.
|
||||
@@ -147,7 +147,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: HxScanOrchestrator_
|
||||
- _Depends: 5.2_
|
||||
|
||||
- [ ] 6.2 Map stage events and identifiers onto HarnessX hooks
|
||||
- [x] 6.2 Map stage events and identifiers onto HarnessX hooks
|
||||
- Map the existing stage-start/stage-end events to the HarnessX step-level hooks and the scan identifier to the HarnessX run identifier without losing trace continuity.
|
||||
- Observable completion: a HarnessX-backed run preserves trace continuity with stage events mapped to step hooks and the scan id mapped to the run id, verified by inspecting the emitted trace.
|
||||
- _Requirements: 1.7_
|
||||
@@ -162,7 +162,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: HxScanOrchestrator_
|
||||
- _Depends: 6.1_
|
||||
|
||||
- [ ] 6.4 (P) Add the LLM-judge verifier with a structural fallback
|
||||
- [x] 6.4 (P) Add the LLM-judge verifier with a structural fallback
|
||||
- Produce verification verdicts using the HarnessX llm-judge helpers when the extra is present, mapping the strict-JSON verdict to a verification result.
|
||||
- Retain the existing structural verifier (which rejects on unknown reachability) verbatim as the zero-dependency fallback.
|
||||
- Observable completion: with the extra present the judge produces verdicts and with it absent the structural verifier produces equivalent verdicts, proven by a paired verifier test.
|
||||
@@ -170,7 +170,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: VerifyProcessor_
|
||||
- _Depends: 5.2_
|
||||
|
||||
- [ ] 6.5 Fall back gracefully when a HarnessX capability is unavailable
|
||||
- [x] 6.5 Fall back gracefully when a HarnessX capability is unavailable
|
||||
- When a HarnessX-backed stage is enabled but its capability is unavailable at runtime, fall back to the existing equivalent stage and record the degradation in the scan manifest.
|
||||
- Observable completion: requesting a HarnessX stage whose capability is missing runs the existing equivalent and records a degradation entry in the manifest, proven by a fallback test.
|
||||
- _Requirements: 1.8_
|
||||
|
||||
Reference in New Issue
Block a user