Wire HarnessX as an optional, SHA-pinned extra with a lazy capability guard,
keeping the core install zero-dependency.
- pyproject: add [project.optional-dependencies] harnessx pinned to commit
274c512 (Darwin-Agent/HarnessX). Core dependencies stays []. uv_build emits
`Requires-Dist: harnessx @ git+...@<sha> ; extra == 'harnessx'` +
`Provides-Extra: harnessx` (verified via uv build); release ships to GitHub
Releases, not PyPI, so the direct-reference is fine.
- harness_ext.py: the capability seam. has_harnessx() probes via find_spec
(never imports harnessx); require_harnessx() imports lazily on demand and
raises HarnessXUnavailableError with install guidance when absent. Every
Phase 3+ HarnessX path routes through here so heavy imports stay cold by default.
- uv.lock refreshed: the universal lock now resolves the harnessx tree (149 pkgs)
but `uv sync --group dev` installs only 57 — harnessx is NOT importable in the
default/CI venv (verified), so CI stays lean and never re-locks.
- Tests: core deps empty; the extra is a 40-hex SHA-pinned git URL; importing
openultrasast.cli does not import harnessx; has_harnessx probes without
importing; require_harnessx raises with guidance when absent.
98 tests pass; ruff + mypy clean; detection gate 93.6%/0% intact. Spec task 5
(5.1 + 5.2) checked off; next is task 6 (build_sast hunter sub-harness).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The dagger job failed pulling python:3.11-bookworm from Docker Hub with a 520
(transient auth/CDN outage) — the residual risk noted in the Phase 3 dependency
audit. The earlier hardening pre-pulled only the dagger engine image
(registry.dagger.io); the base-image pull happens inside the engine via
client.container().from_(...), which the host pre-pull cannot cover.
Wrap the whole dagger pipeline (_run_checks) in a 3-attempt retry that re-runs on
transient registry/pull markers (Docker Hub 5xx/429, oauth/authorize failures,
TransportQueryError, i/o timeout, connection reset, context deadline). Permanent
failures (lint/type/test) still fail immediately on the first attempt.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Run and record the HarnessX dependency dry-run audit and assess blueprint
alignment with the author's DocuHarnessX, to steer OpenUltraSAST's HarnessX
adoption from opencode/claude-code.
- dependency-audit.md: records the realized transitive graph (118 packages from
30 mandatory HarnessX deps). Key finding: every heavy backend (docker/e2b/
playwright/office-PDF) is lazily imported and the gateway is a separate dist, so
the tree is an install-time (not runtime) cost; ~55-60 packages are dead weight
for SAST. Decision: ACCEPT as the optional SHA-pinned extra openultrasast[harnessx]
(core stays zero-dep; lazy heavy imports; matches DocuHarnessX precedent), with a
fully-specified vendored lean subset as the triggered fallback and upstream
dependency-extras as the preferred long-term slimming path. Records the
security-tool residual risks (browser binary, docker socket, web server, CVE
surface) and their mitigations.
- blueprint-alignment.md: DocuHarnessX <-> OpenUltraSAST assessment. They share the
spine (one composition factory, model bound in CLI via ModelConfig.agentic,
content-neutral processors over typed slots, project-loaded domain/"ontology"
seam, fail-closed LLM-judge firewall, Kiro fresh-subagent steering). DocuHarnessX
realizes "self-improving multi-agent" via the firewall + claude-code/opencode
steering loop, NOT MetaAgent.evolve (Train is future there); OpenUltraSAST builds
the evolve loop on the same substrate. Three principled divergences (hybrid runtime
confining HarnessX to the hunter pool; evolve reward/feasibility split + ledger;
zero-dep core with opt-in extra) with reconciliation, plus a concrete Phase 3
blueprint mirroring DocuHarnessX.
Checks off spec task 5.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.
- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
(keep/shadow/loosen) written to rule_policy_delta.json; the inert
next_improvement_candidate stub is now rule-aware ("rule:<id>" or
"language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
(empty -> byte-identical; an entry overlays a rule's status/floors at load);
read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
finding's effective reachability multiplier in the score
(fp_reachability_overrides + scoring reachability_override) instead of deleting
the rule, so recall stays high while the score stops over-penalizing.
93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Externalize detection rules to versioned TOML and remove rule-local severity, so
rules detect and the central CWE policy decides severity — fixing "rules too
influential / not centrally managed". Zero-dependency; detection is byte-identical
(benchmark recall 44/47 = 93.6%, FP 0% unchanged) and the 90/10 gate holds.
- RulesetStore (ruleset/store.py + ruleset/<lang>/rules.toml): PatternRule as data
(rule_id, title, languages, cwe, tags, pattern, status, min_evidence_level,
precision_estimate, version) with NO severity; load_ruleset (sorted, with
optional rule_policy.json ledger overlay) and a stdlib TOML write_ruleset. The
32 rules were generated from the prior PATTERN_RULES with a verified exact
round-trip (pattern/cwe/languages/tags/title).
- findings.py: drop the hardcoded PatternRule class + PATTERN_RULES tuple; load the
default ruleset from TOML; quick_scan_findings(root, targets, rankings, ruleset,
policy) resolves each finding's severity from policy via resolve_severity
(SEVERITY_LABEL maps 0-5 -> string) and tags findings with their rule status.
- Three-status volume control: StaticFinding gains status (default "enabled");
disabled rules never fire; shadow findings are partitioned out of the report and
score in the CLI (written to shadow_findings.json) — no shadow rules by default.
- Emission floors (RulesetConfig.min_emit_priority / min_emit_precision, default
0.0 = emit all) replace "every match is a finding".
- CLI loads the ruleset as a stage and uses it for the fail-loud policy check,
detection, hunter pool, and the score CWE map. Wheel ships the ruleset TOML.
85 tests pass; ruff + mypy clean. Spec tasks 3.1-3.4 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CI: the dagger job repeatedly failed (and once hung 10 min) pulling the engine
image from registry.dagger.io. Pin the dagger CLI to 0.21.7 (matching dagger-io
and the SDK's CLI_VERSION so the engine tag is deterministic), pre-pull
registry.dagger.io/engine:v0.21.7 with a dependency-free retry loop (5 attempts,
90s/pull, 20s backoff) so the action's `docker image inspect` finds it cached and
skips its own flaky pull, and add a 20-minute job timeout so a stalled pull can
no longer hang the run. The canonical `checks` job is unchanged.
Docs: add a "Project scores and central CWE policy" README section documenting
the implemented Phase 1 — central verycode CWE policy, policy-governed severity
(one CWE = one severity), the fail-loud invariant, the 0-100 project score and
two-condition advisory-first gate, the score.json/manifest artifacts, and the
CWE-120 -> CWE-121 remap.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Spec-driven plan to (1) adopt HarnessX (Darwin-Agent/HarnessX) directly where it
pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness
self-improvement loop — and migrate scan flows onto it stage-by-stage; (2) make
detection rules versioned, centrally-governed data whose severity comes from a
vendored verycode CWE policy (CWE_Score.tsv) instead of the rule itself, fixing
"rules too influential / not centrally managed"; (3) adapt rules/policy inside
HarnessX's MetaAgent.evolve loop under bounded levers with replay/novelty/
validator gates; (4) make benchmarks actionable per-rule signals with a hard
accept gate and byte-for-byte revert; (5) compute a 0-100 verycode-style project
score with a two-condition CI gate.
Key decisions captured in research.md: HarnessX is an LLM-agent harness (not a
generic stage pipeline), so it is adopted only on the agentic plane behind an
optional SHA-pinned extra while the governance/scoring/benchmark planes stay
zero-dependency; the >=90% recall / <10% FP detection gate is a hard constraint
separate from the score reward and must hold at every phase; CWE-120 is unmapped
in verycode so C/C++ buffer rules remap to CWE-121 (severity 5).
Artifacts: spec.json (phase tasks-generated, approvals pending), requirements.md
(7 requirements, EARS), design.md (three-plane architecture), tasks.md (4
phases), research.md (D1-D9 decisions). Awaiting human approval per the kiro
3-phase gate; not yet implemented.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`files: dist/*` also picked up the `dist/.gitignore` uv writes, attaching a
stray `default.gitignore` asset to the release. Restrict to dist/*.whl and
dist/*.tar.gz.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:
- New `calibrate` stage (after rank, before findings) loads a persistent ledger
of scoped false-positive learnings and demotes every previously-rejected scope
(`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into
`.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
rejections can demote a scope below the natural ranking minimum.
Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.
Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- OpenUltraSAST is driven from the command line (ousast CLI, or OpenCode invoked
on the repo running the harness for you); show both.
- Mark the kiro-* skills and AGENTS.md as maintainer-only SDLC tooling, not part
of using OpenUltraSAST; drop them from the user-facing skill list.
- State that OpenCode is the agent steering the HarnessX self-improving cycle
today (reads verifier/benchmark signals, applies scoped calibration via the
triage skill, re-runs the gate); autonomous in-pipeline feedback is the next
milestone.
- Remove a duplicated Fusion section.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expand the README with: how to run the per-language benchmarks and the latest
measured results (C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0% false positives); the OpenCode skills, the ultra
(OpenUltraCode) fix lifecycle, and fusion deepening; how a false positive is
eliminated via the evidence-ladder verifier and scoped calibration; how a false
negative becomes a benchmark calibration record routed to a next-improvement
candidate; and how the HarnessX self-improving cycle is realized. Maturity is
marked (implemented / primitive / roadmap) to stay accurate to the current gate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add the Kiro-style agentic SDLC tooling used to drive this project: the
.opencode/skills/kiro-* skill set (discovery, spec init/requirements/design/
tasks, validation, implementation, review, steering), the .kiro/settings
templates, and AGENTS.md describing the workflow. Update the spec docs
(assessment/design/requirements/tasks) to current state.
Scope tests/test_opencode_skills.py to the project's own openultrasast-*
skills so the third-party kiro-* tooling (which uses a different frontmatter
convention) no longer breaks the frontmatter check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CI runs `ruff format --check .` and `ruff check .` over the whole repo. The
benchmark fixtures are intentionally vulnerable, unformatted sample code (e.g.
a `% uid` SQL injection that ruff flags as UP031), and reformatting them would
shift the ground-truth line numbers the manifests depend on. Exclude
benchmarks/fixtures from ruff so lint/format only cover real project sources.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Rebuild the quick-scan detection engine and ground-truth benchmarks to hit
>=90% recall / <10% false positives for C/C++, Python, JavaScript, and Java.
Engine (findings.py, preprocess.py):
- Scope each PatternRule by language (fixes cross-language false positives such
as the Python eval/exec rule firing on JS .exec()).
- Add JavaScript/TypeScript rules (command exec, eval/Function, reflected XSS,
SQLi, path traversal, SSRF, weak hash, deserialization); execFile/spawn-argv
excluded as safe alternatives.
- Add Java rules (Runtime exec, SQL string concatenation, weak hash, readObject)
and a Groovy server-side template rule (yieldUnescaped of a bound variable).
- Broaden C/C++ and Python coverage; precision guards for format strings and
parameterised vs concatenated SQL; #-comment skipping; CWE in rationale.
- Register .java/.groovy/.tpl source extensions.
Benchmarks:
- Add realistic corpora modeled on Damn Vulnerable C/C++, PyGoat/DVPWA, and
NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored
verbatim (MIT, attribution in LICENSE/NOTICE.md). Each manifest carries
complete, independently-labeled ground truth including genuinely
hard-to-detect vulns (integer overflow, second-order SQLi, prototype
pollution) so recall is measured honestly, plus safe-API files for precision.
- Fix pre-existing smoke-manifest ground-truth errors (Python/JS line offsets,
missing C/C++ strcpy entry, Java SQLi line).
benchmark.py:
- Fix sink matching to normalise the expected sink consistently so sinks with
underscores (e.g. render_template_string) can match.
Tests:
- Add tests/test_detection_benchmarks.py enforcing the >=90% recall / <10% FP
goal per language and overall.
Measured: C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0.0% false-positive rate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
