Review rounds 3-5 remediations (Req 5, 7).
- ousast improve resolves candidates from the target's calibration dir,
then the working directory's (where `mechanisms export` writes), and
admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
where a scan of that target reads; --mechanism-candidates,
--mechanism-store, --no-mechanisms; --dry-run redirects the store to
scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
mechanism edits; reverted mechanism keys are blocked outright; a
no-gain admission is rejected as mechanism_no_gain; a retraction may
name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
containment and label rules; mechanisms export defaults to the
candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
export -> improve -> scan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.1 (Req 5.1, 5.2, 5.5).
- validator: MechanismEdit (admit | retract by record id) under the
`mechanisms` lever; validate_mechanism accepts only records from the
exporter candidate set with origin corpus, a closed guard and source
kinds, and identifier-only shape text; free-form shapes are rejected.
- apply_mechanism_edits appends the record (admit) or a retracted
tombstone (retract) so the log stays append-only; StoreSnapshot.restore
reverts byte for byte. MechanismStore.load folds tombstones away.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Stand up the deterministic self-improvement loop over the ruleset/policy
governance data. Zero-dependency and fully tested offline; detection gate intact.
- 7.1/7.2 bounded levers + validator (improve/validator.py): two levers (rule,
policy); typed edits make pattern-text and 0-5 severity changes structurally
impossible; EvolveValidator rejects bad status, unknown rule_ids (loop cannot add
rules), enabled->disabled jumps (shadow staging mandatory), unresolved CWEs, and
out-of-bounds K/MIN_SCORE, raising StrictValidationError.
- 7.3 novelty + replay + journal (improve/journal.py): a JSON round journal records
every edit/lever/outcome; reverted edits are blocked from re-proposal (novelty);
a replay smoke gate boots the candidate ruleset before re-benchmarking.
- 7.4 benchmark deltas -> rule_signals.json (recall side: miss; precision side: fp).
- 7.5 hard acceptance gate + byte-for-byte revert (improve/evolve.py): a round is
accepted only if recall >= 90% AND FP < 10% AND project score did not regress AND
no matched-finding regression; otherwise the ledger is never written (byte-for-byte
revert). run_round/run_improvement drive rounds to convergence.
- 7.6 auto-shadow precision-draggers (enabled rules with FPs -> shadow, never
delete), nominate coverage gaps via rule_signals, and keep the project score as the
reward separate from the recall/FP feasibility gate (the loop cannot raise the
score by shadowing rules that breach the gate — proven by the revert test).
Design note: the loop's safety machinery is deterministic and benchmark-driven
(the proposer is the per-rule signal); the LLM MetaAgent.evolve is the optional
richer proposer that plugs into this same validated/gated machinery when the
HarnessX extra is present (offline-untestable). This mirrors how DocuHarnessX
realizes self-improvement.
9 new tests (validator, proposer/signals, accept, byte-for-byte revert, convergence);
118 pass total, 2 construction smokes skip without the extra; ruff + mypy clean;
detection gate 93.6%/0% intact. Tasks 7.1-7.6 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
