Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).
- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
The provider name is validated before any import, so an unknown choice fails with
a clear ValueError even when the extra is absent; each provider reads its own
standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
provider, budgets, fallback/degradation).
Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.
137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expose the bounded, gated self-improvement loop as a first-class subcommand so it
can be steered from the command line / opencode.
- `ousast improve <manifest>` resolves the target from the benchmark manifest,
runs the loop to convergence, and writes accepted edits to the loop-owned ledger
at <target>/.openultrasast/calibration/rule_policy.json — exactly where `scan`
and `benchmark` read it, so a subsequent scan of that target automatically
picks up the improved ruleset (the loop closes end-to-end).
- Flags: --max-rounds, --recall-floor, --fp-ceiling, --ledger, --journal,
--ruleset-dir, and --dry-run (runs against a throwaway ledger, touches nothing).
- Per-round output: reason, recall/fp/score deltas, and the edits applied.
- tests/test_cli_improve.py: accept path writes the loop-owned ledger (noisy rule
auto-shadowed), --dry-run never touches it, missing manifest fails loudly.
- README: quickstart entry + a "Rule-level loop: ousast improve" subsection
documenting the bounded levers, hard acceptance gate, and the HarnessX MetaAgent
as the optional proposer over the same gated machinery.
131 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make the >=90% recall / <10% FP detection gate a first-class, score-independent
CI step with non-regression guards.
- openultrasast.gate: canonical corpus map + DetectionMetrics/DetectionGateVerdict,
evaluate_corpus/detection_gate (no score input — cannot be unblocked by score),
merge_gate (ANDs the hard detection gate with the advisory-first score gate),
within_tolerance, and a `python -m openultrasast.gate` entrypoint that exits
non-zero on a breach. Wired as an explicit step in both the GitHub and Dagger
pipelines.
- 8.1: pipeline tests prove a high/passing score cannot unblock a recall-breaching
change; the gate fails below 90% recall or at/above 10% FP.
- 8.2: tests/golden/detection_baseline.json + a byte-identical baseline test for
the default ruleset / empty ledger (44/47 = 93.62% recall, 0% FP).
- 8.3: within_tolerance band logic + a capability-gated HarnessX tolerance test.
- 8.4: zero-dependency guard — governance/scoring/benchmark planes never import
HarnessX (sys.modules check + module-scope import scan).
- test_detection_benchmarks now sources the corpus/aggregation from the gate module.
128 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stand up the deterministic self-improvement loop over the ruleset/policy
governance data. Zero-dependency and fully tested offline; detection gate intact.
- 7.1/7.2 bounded levers + validator (improve/validator.py): two levers (rule,
policy); typed edits make pattern-text and 0-5 severity changes structurally
impossible; EvolveValidator rejects bad status, unknown rule_ids (loop cannot add
rules), enabled->disabled jumps (shadow staging mandatory), unresolved CWEs, and
out-of-bounds K/MIN_SCORE, raising StrictValidationError.
- 7.3 novelty + replay + journal (improve/journal.py): a JSON round journal records
every edit/lever/outcome; reverted edits are blocked from re-proposal (novelty);
a replay smoke gate boots the candidate ruleset before re-benchmarking.
- 7.4 benchmark deltas -> rule_signals.json (recall side: miss; precision side: fp).
- 7.5 hard acceptance gate + byte-for-byte revert (improve/evolve.py): a round is
accepted only if recall >= 90% AND FP < 10% AND project score did not regress AND
no matched-finding regression; otherwise the ledger is never written (byte-for-byte
revert). run_round/run_improvement drive rounds to convergence.
- 7.6 auto-shadow precision-draggers (enabled rules with FPs -> shadow, never
delete), nominate coverage gaps via rule_signals, and keep the project score as the
reward separate from the recall/FP feasibility gate (the loop cannot raise the
score by shadowing rules that breach the gate — proven by the revert test).
Design note: the loop's safety machinery is deterministic and benchmark-driven
(the proposer is the per-rule signal); the LLM MetaAgent.evolve is the optional
richer proposer that plugs into this same validated/gated machinery when the
HarnessX extra is present (offline-untestable). This mirrors how DocuHarnessX
realizes self-improvement.
9 new tests (validator, proposer/signals, accept, byte-for-byte revert, convergence);
118 pass total, 2 construction smokes skip without the extra; ruff + mypy clean;
detection gate 93.6%/0% intact. Tasks 7.1-7.6 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adopt HarnessX at the hunter loop and verifier, capability-gated so the default
zero-dependency scan is byte-identical and CI (no extra) exercises the fallback.
- 6.1/6.2 build_sast() (single, lazy HarnessX composition site in harness_ext.py,
mirrors DocuHarnessX make_docgen) + HxScanOrchestrator (hunter_harness.py): a
drop-in for run_hunter_pool that runs one bounded HarnessX agent task per ranked
target (ModelConfig(main=AnthropicProvider).agentic(build_sast()).run(BaseTask)),
parses findings from a JSON structured-output contract, returns the existing
HunterPoolResult, maps scan_id->session_id and emits per-target exit_reason/cost/
steps to the outer trace.
- 6.4 verify_judge.verify_findings_dispatch: a drop-in superset of
verification.verify_findings — structural verifier verbatim by default (rejects
unknown reachability), HarnessX llm-judge (build_judge_prompt/parse_judge_response)
when the extra + a verifier model are present; verdict mapped to VerificationResult
evidence-gated (plausible>=0.6 accepted; unsupported/refused rejected; infra
failure never auto-rejects).
- 6.5 cli capability gate (hx_hunter/hx_verify): selects HxScanOrchestrator /
judge only in standard mode with a configured model AND has_harnessx(); otherwise
falls back and records a {stage,reason:harnessx_extra_unavailable} degradation in
the manifest (new append-only write_manifest degradations kwarg). With no model
configured the manifest stays byte-identical.
- pyproject: mypy override ignore_missing_imports for harnessx.* (absent in the
type-check env).
Verified the HarnessX path constructs against the real extra in a throwaway venv
(build_sast -> HarnessConfig; orchestrator -> Harness; build_judge_prompt/
parse_judge_response/verdict mapping) — only the live agent.run()/provider.complete()
LLM calls are offline-untestable and stay gated. 109 tests pass (2 construction
smokes skip without the extra), ruff + mypy clean, detection gate 93.6%/0% intact.
Tasks 6.1/6.2/6.4/6.5 done; 6.3 (deterministic stages as HarnessX processors) deferred.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wire HarnessX as an optional, SHA-pinned extra with a lazy capability guard,
keeping the core install zero-dependency.
- pyproject: add [project.optional-dependencies] harnessx pinned to commit
274c512 (Darwin-Agent/HarnessX). Core dependencies stays []. uv_build emits
`Requires-Dist: harnessx @ git+...@<sha> ; extra == 'harnessx'` +
`Provides-Extra: harnessx` (verified via uv build); release ships to GitHub
Releases, not PyPI, so the direct-reference is fine.
- harness_ext.py: the capability seam. has_harnessx() probes via find_spec
(never imports harnessx); require_harnessx() imports lazily on demand and
raises HarnessXUnavailableError with install guidance when absent. Every
Phase 3+ HarnessX path routes through here so heavy imports stay cold by default.
- uv.lock refreshed: the universal lock now resolves the harnessx tree (149 pkgs)
but `uv sync --group dev` installs only 57 — harnessx is NOT importable in the
default/CI venv (verified), so CI stays lean and never re-locks.
- Tests: core deps empty; the extra is a 40-hex SHA-pinned git URL; importing
openultrasast.cli does not import harnessx; has_harnessx probes without
importing; require_harnessx raises with guidance when absent.
98 tests pass; ruff + mypy clean; detection gate 93.6%/0% intact. Spec task 5
(5.1 + 5.2) checked off; next is task 6 (build_sast hunter sub-harness).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The dagger job failed pulling python:3.11-bookworm from Docker Hub with a 520
(transient auth/CDN outage) — the residual risk noted in the Phase 3 dependency
audit. The earlier hardening pre-pulled only the dagger engine image
(registry.dagger.io); the base-image pull happens inside the engine via
client.container().from_(...), which the host pre-pull cannot cover.
Wrap the whole dagger pipeline (_run_checks) in a 3-attempt retry that re-runs on
transient registry/pull markers (Docker Hub 5xx/429, oauth/authorize failures,
TransportQueryError, i/o timeout, connection reset, context deadline). Permanent
failures (lint/type/test) still fail immediately on the first attempt.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Run and record the HarnessX dependency dry-run audit and assess blueprint
alignment with the author's DocuHarnessX, to steer OpenUltraSAST's HarnessX
adoption from opencode/claude-code.
- dependency-audit.md: records the realized transitive graph (118 packages from
30 mandatory HarnessX deps). Key finding: every heavy backend (docker/e2b/
playwright/office-PDF) is lazily imported and the gateway is a separate dist, so
the tree is an install-time (not runtime) cost; ~55-60 packages are dead weight
for SAST. Decision: ACCEPT as the optional SHA-pinned extra openultrasast[harnessx]
(core stays zero-dep; lazy heavy imports; matches DocuHarnessX precedent), with a
fully-specified vendored lean subset as the triggered fallback and upstream
dependency-extras as the preferred long-term slimming path. Records the
security-tool residual risks (browser binary, docker socket, web server, CVE
surface) and their mitigations.
- blueprint-alignment.md: DocuHarnessX <-> OpenUltraSAST assessment. They share the
spine (one composition factory, model bound in CLI via ModelConfig.agentic,
content-neutral processors over typed slots, project-loaded domain/"ontology"
seam, fail-closed LLM-judge firewall, Kiro fresh-subagent steering). DocuHarnessX
realizes "self-improving multi-agent" via the firewall + claude-code/opencode
steering loop, NOT MetaAgent.evolve (Train is future there); OpenUltraSAST builds
the evolve loop on the same substrate. Three principled divergences (hybrid runtime
confining HarnessX to the hunter pool; evolve reward/feasibility split + ledger;
zero-dep core with opt-in extra) with reconciliation, plus a concrete Phase 3
blueprint mirroring DocuHarnessX.
Checks off spec task 5.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.
- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
(keep/shadow/loosen) written to rule_policy_delta.json; the inert
next_improvement_candidate stub is now rule-aware ("rule:<id>" or
"language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
(empty -> byte-identical; an entry overlays a rule's status/floors at load);
read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
finding's effective reachability multiplier in the score
(fp_reachability_overrides + scoring reachability_override) instead of deleting
the rule, so recall stays high while the score stops over-penalizing.
93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Externalize detection rules to versioned TOML and remove rule-local severity, so
rules detect and the central CWE policy decides severity — fixing "rules too
influential / not centrally managed". Zero-dependency; detection is byte-identical
(benchmark recall 44/47 = 93.6%, FP 0% unchanged) and the 90/10 gate holds.
- RulesetStore (ruleset/store.py + ruleset/<lang>/rules.toml): PatternRule as data
(rule_id, title, languages, cwe, tags, pattern, status, min_evidence_level,
precision_estimate, version) with NO severity; load_ruleset (sorted, with
optional rule_policy.json ledger overlay) and a stdlib TOML write_ruleset. The
32 rules were generated from the prior PATTERN_RULES with a verified exact
round-trip (pattern/cwe/languages/tags/title).
- findings.py: drop the hardcoded PatternRule class + PATTERN_RULES tuple; load the
default ruleset from TOML; quick_scan_findings(root, targets, rankings, ruleset,
policy) resolves each finding's severity from policy via resolve_severity
(SEVERITY_LABEL maps 0-5 -> string) and tags findings with their rule status.
- Three-status volume control: StaticFinding gains status (default "enabled");
disabled rules never fire; shadow findings are partitioned out of the report and
score in the CLI (written to shadow_findings.json) — no shadow rules by default.
- Emission floors (RulesetConfig.min_emit_priority / min_emit_precision, default
0.0 = emit all) replace "every match is a finding".
- CLI loads the ruleset as a stage and uses it for the fail-loud policy check,
detection, hunter pool, and the score CWE map. Wheel ships the ruleset TOML.
85 tests pass; ruff + mypy clean. Spec tasks 3.1-3.4 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CI: the dagger job repeatedly failed (and once hung 10 min) pulling the engine
image from registry.dagger.io. Pin the dagger CLI to 0.21.7 (matching dagger-io
and the SDK's CLI_VERSION so the engine tag is deterministic), pre-pull
registry.dagger.io/engine:v0.21.7 with a dependency-free retry loop (5 attempts,
90s/pull, 20s backoff) so the action's `docker image inspect` finds it cached and
skips its own flaky pull, and add a 20-minute job timeout so a stalled pull can
no longer hang the run. The canonical `checks` job is unchanged.
Docs: add a "Project scores and central CWE policy" README section documenting
the implemented Phase 1 — central verycode CWE policy, policy-governed severity
(one CWE = one severity), the fail-loud invariant, the 0-100 project score and
two-condition advisory-first gate, the score.json/manifest artifacts, and the
CWE-120 -> CWE-121 remap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Spec-driven plan to (1) adopt HarnessX (Darwin-Agent/HarnessX) directly where it
pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness
self-improvement loop — and migrate scan flows onto it stage-by-stage; (2) make
detection rules versioned, centrally-governed data whose severity comes from a
vendored verycode CWE policy (CWE_Score.tsv) instead of the rule itself, fixing
"rules too influential / not centrally managed"; (3) adapt rules/policy inside
HarnessX's MetaAgent.evolve loop under bounded levers with replay/novelty/
validator gates; (4) make benchmarks actionable per-rule signals with a hard
accept gate and byte-for-byte revert; (5) compute a 0-100 verycode-style project
score with a two-condition CI gate.
Key decisions captured in research.md: HarnessX is an LLM-agent harness (not a
generic stage pipeline), so it is adopted only on the agentic plane behind an
optional SHA-pinned extra while the governance/scoring/benchmark planes stay
zero-dependency; the >=90% recall / <10% FP detection gate is a hard constraint
separate from the score reward and must hold at every phase; CWE-120 is unmapped
in verycode so C/C++ buffer rules remap to CWE-121 (severity 5).
Artifacts: spec.json (phase tasks-generated, approvals pending), requirements.md
(7 requirements, EARS), design.md (three-plane architecture), tasks.md (4
phases), research.md (D1-D9 decisions). Awaiting human approval per the kiro
3-phase gate; not yet implemented.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`files: dist/*` also picked up the `dist/.gitignore` uv writes, attaching a
stray `default.gitignore` asset to the release. Restrict to dist/*.whl and
dist/*.tar.gz.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:
- New `calibrate` stage (after rank, before findings) loads a persistent ledger
of scoped false-positive learnings and demotes every previously-rejected scope
(`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into
`.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
rejections can demote a scope below the natural ranking minimum.
Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.
Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- OpenUltraSAST is driven from the command line (ousast CLI, or OpenCode invoked
on the repo running the harness for you); show both.
- Mark the kiro-* skills and AGENTS.md as maintainer-only SDLC tooling, not part
of using OpenUltraSAST; drop them from the user-facing skill list.
- State that OpenCode is the agent steering the HarnessX self-improving cycle
today (reads verifier/benchmark signals, applies scoped calibration via the
triage skill, re-runs the gate); autonomous in-pipeline feedback is the next
milestone.
- Remove a duplicated Fusion section.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expand the README with: how to run the per-language benchmarks and the latest
measured results (C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0% false positives); the OpenCode skills, the ultra
(OpenUltraCode) fix lifecycle, and fusion deepening; how a false positive is
eliminated via the evidence-ladder verifier and scoped calibration; how a false
negative becomes a benchmark calibration record routed to a next-improvement
candidate; and how the HarnessX self-improving cycle is realized. Maturity is
marked (implemented / primitive / roadmap) to stay accurate to the current gate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add the Kiro-style agentic SDLC tooling used to drive this project: the
.opencode/skills/kiro-* skill set (discovery, spec init/requirements/design/
tasks, validation, implementation, review, steering), the .kiro/settings
templates, and AGENTS.md describing the workflow. Update the spec docs
(assessment/design/requirements/tasks) to current state.
Scope tests/test_opencode_skills.py to the project's own openultrasast-*
skills so the third-party kiro-* tooling (which uses a different frontmatter
convention) no longer breaks the frontmatter check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CI runs `ruff format --check .` and `ruff check .` over the whole repo. The
benchmark fixtures are intentionally vulnerable, unformatted sample code (e.g.
a `% uid` SQL injection that ruff flags as UP031), and reformatting them would
shift the ground-truth line numbers the manifests depend on. Exclude
benchmarks/fixtures from ruff so lint/format only cover real project sources.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Rebuild the quick-scan detection engine and ground-truth benchmarks to hit
>=90% recall / <10% false positives for C/C++, Python, JavaScript, and Java.
Engine (findings.py, preprocess.py):
- Scope each PatternRule by language (fixes cross-language false positives such
as the Python eval/exec rule firing on JS .exec()).
- Add JavaScript/TypeScript rules (command exec, eval/Function, reflected XSS,
SQLi, path traversal, SSRF, weak hash, deserialization); execFile/spawn-argv
excluded as safe alternatives.
- Add Java rules (Runtime exec, SQL string concatenation, weak hash, readObject)
and a Groovy server-side template rule (yieldUnescaped of a bound variable).
- Broaden C/C++ and Python coverage; precision guards for format strings and
parameterised vs concatenated SQL; #-comment skipping; CWE in rationale.
- Register .java/.groovy/.tpl source extensions.
Benchmarks:
- Add realistic corpora modeled on Damn Vulnerable C/C++, PyGoat/DVPWA, and
NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored
verbatim (MIT, attribution in LICENSE/NOTICE.md). Each manifest carries
complete, independently-labeled ground truth including genuinely
hard-to-detect vulns (integer overflow, second-order SQLi, prototype
pollution) so recall is measured honestly, plus safe-API files for precision.
- Fix pre-existing smoke-manifest ground-truth errors (Python/JS line offsets,
missing C/C++ strcpy entry, Java SQLi line).
benchmark.py:
- Fix sink matching to normalise the expected sink consistently so sinks with
underscores (e.g. render_template_string) can match.
Tests:
- Add tests/test_detection_benchmarks.py enforcing the >=90% recall / <10% FP
goal per language and overall.
Measured: C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0.0% false-positive rate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
