Spec-driven plan to (1) adopt HarnessX (Darwin-Agent/HarnessX) directly where it
pays off — the LLM hunter loop, the LLM-judge verifier, and a meta-harness
self-improvement loop — and migrate scan flows onto it stage-by-stage; (2) make
detection rules versioned, centrally-governed data whose severity comes from a
vendored verycode CWE policy (CWE_Score.tsv) instead of the rule itself, fixing
"rules too influential / not centrally managed"; (3) adapt rules/policy inside
HarnessX's MetaAgent.evolve loop under bounded levers with replay/novelty/
validator gates; (4) make benchmarks actionable per-rule signals with a hard
accept gate and byte-for-byte revert; (5) compute a 0-100 verycode-style project
score with a two-condition CI gate.
Key decisions captured in research.md: HarnessX is an LLM-agent harness (not a
generic stage pipeline), so it is adopted only on the agentic plane behind an
optional SHA-pinned extra while the governance/scoring/benchmark planes stay
zero-dependency; the >=90% recall / <10% FP detection gate is a hard constraint
separate from the score reward and must hold at every phase; CWE-120 is unmapped
in verycode so C/C++ buffer rules remap to CWE-121 (severity 5).
Artifacts: spec.json (phase tasks-generated, approvals pending), requirements.md
(7 requirements, EARS), design.md (three-plane architecture), tasks.md (4
phases), research.md (D1-D9 decisions). Awaiting human approval per the kiro
3-phase gate; not yet implemented.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:
- New `calibrate` stage (after rank, before findings) loads a persistent ledger
of scoped false-positive learnings and demotes every previously-rejected scope
(`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into
`.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
rejections can demote a scope below the natural ranking minimum.
Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.
Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add the Kiro-style agentic SDLC tooling used to drive this project: the
.opencode/skills/kiro-* skill set (discovery, spec init/requirements/design/
tasks, validation, implementation, review, steering), the .kiro/settings
templates, and AGENTS.md describing the workflow. Update the spec docs
(assessment/design/requirements/tasks) to current state.
Scope tests/test_opencode_skills.py to the project's own openultrasast-*
skills so the third-party kiro-* tooling (which uses a different frontmatter
convention) no longer breaks the frontmatter check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
