mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 04:29:27 +00:00
feat(improve): Phase 3 task 7 — bounded, gated self-improvement loop
Stand up the deterministic self-improvement loop over the ruleset/policy governance data. Zero-dependency and fully tested offline; detection gate intact. - 7.1/7.2 bounded levers + validator (improve/validator.py): two levers (rule, policy); typed edits make pattern-text and 0-5 severity changes structurally impossible; EvolveValidator rejects bad status, unknown rule_ids (loop cannot add rules), enabled->disabled jumps (shadow staging mandatory), unresolved CWEs, and out-of-bounds K/MIN_SCORE, raising StrictValidationError. - 7.3 novelty + replay + journal (improve/journal.py): a JSON round journal records every edit/lever/outcome; reverted edits are blocked from re-proposal (novelty); a replay smoke gate boots the candidate ruleset before re-benchmarking. - 7.4 benchmark deltas -> rule_signals.json (recall side: miss; precision side: fp). - 7.5 hard acceptance gate + byte-for-byte revert (improve/evolve.py): a round is accepted only if recall >= 90% AND FP < 10% AND project score did not regress AND no matched-finding regression; otherwise the ledger is never written (byte-for-byte revert). run_round/run_improvement drive rounds to convergence. - 7.6 auto-shadow precision-draggers (enabled rules with FPs -> shadow, never delete), nominate coverage gaps via rule_signals, and keep the project score as the reward separate from the recall/FP feasibility gate (the loop cannot raise the score by shadowing rules that breach the gate — proven by the revert test). Design note: the loop's safety machinery is deterministic and benchmark-driven (the proposer is the per-rule signal); the LLM MetaAgent.evolve is the optional richer proposer that plugs into this same validated/gated machinery when the HarnessX extra is present (offline-untestable). This mirrors how DocuHarnessX realizes self-improvement. 9 new tests (validator, proposer/signals, accept, byte-for-byte revert, convergence); 118 pass total, 2 construction smokes skip without the extra; ruff + mypy clean; detection gate 93.6%/0% intact. Tasks 7.1-7.6 checked off. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
ef59c0ae3d
commit
6e87a58465
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "harnessx-self-improving-rulesets",
|
||||
"created_at": "2026-06-22T10:07:18Z",
|
||||
"updated_at": "2026-06-22T15:05:00Z",
|
||||
"updated_at": "2026-06-22T15:35:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -177,15 +177,15 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: HxScanOrchestrator_
|
||||
- _Depends: 6.1, 6.4_
|
||||
|
||||
- [ ] 7. Stand up the bounded self-improvement loop
|
||||
- [ ] 7.1 Add the two bounded edit levers
|
||||
- [x] 7. Stand up the bounded self-improvement loop
|
||||
- [x] 7.1 Add the two bounded edit levers
|
||||
- Expose exactly two new edit levers: a rule lever covering rule status, evidence floor, and threshold, and a policy lever covering evidence floor, scope, and the score constants.
|
||||
- Observable completion: the loop's valid-lever set includes exactly the rule and policy levers and rejects any edit outside their declared fields, verified by a lever test.
|
||||
- _Requirements: 3.1_
|
||||
- _Boundary: EvolveValidator, improve/evolve.py_
|
||||
- _Depends: 5.2_
|
||||
|
||||
- [ ] 7.2 Enforce the rule/policy edit safety bounds in the validator
|
||||
- [x] 7.2 Enforce the rule/policy edit safety bounds in the validator
|
||||
- Reject any loop edit that modifies detection pattern text and any edit that modifies the authoritative 0-5 CWE severity.
|
||||
- Reject moving a rule directly from enabled to disabled, requiring shadow as a mandatory intermediate staging state.
|
||||
- Assert that each edit parses, that status is one of the three literals, that threshold/evidence-floor changes stay within configured bounds, that an added rule does not duplicate an existing identifier, and that every referenced CWE resolves in policy; raise a rule-change validation error and reject the whole round on any failure.
|
||||
@@ -194,7 +194,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: EvolveValidator_
|
||||
- _Depends: 7.1, 1.1_
|
||||
|
||||
- [ ] 7.3 Add the novelty, replay, and journal gates
|
||||
- [x] 7.3 Add the novelty, replay, and journal gates
|
||||
- Block re-proposing a previously reverted edit that lacks a retry rationale via the novelty gate.
|
||||
- Run a replay smoke gate before any benchmark re-run proving the edited ruleset and policy still load and boot.
|
||||
- Record each rule and policy edit, its lever, and its outcome in the journal so every change is attributable and reversible.
|
||||
@@ -203,14 +203,14 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: EvolveValidator, improve/evolve.py_
|
||||
- _Depends: 7.2_
|
||||
|
||||
- [ ] 7.4 Convert benchmark deltas into loop signals
|
||||
- [x] 7.4 Convert benchmark deltas into loop signals
|
||||
- Convert the per-rule recommendation delta and false-positive learnings into a loop signal file, writing recall-side items as miss signals and precision-side items as verifier-reject or false-positive signals.
|
||||
- Observable completion: a benchmark run produces a signal file whose recall-side entries are miss signals and precision-side entries are verifier-reject/false-positive signals, verified against the documented signal schema.
|
||||
- _Requirements: 4.3_
|
||||
- _Boundary: improve/evolve.py_
|
||||
- _Depends: 4.2, 7.1_
|
||||
|
||||
- [ ] 7.5 Implement the hard acceptance gate with byte-for-byte revert
|
||||
- [x] 7.5 Implement the hard acceptance gate with byte-for-byte revert
|
||||
- After the meta-agent edits rules or policy, re-run the benchmark and accept the round only if aggregate recall is at least 90% AND false-positive rate is under 10% AND the project score did not regress on the held set AND no unpredicted regression occurred.
|
||||
- On any failed acceptance condition, revert the round and restore the rule and policy configuration byte-for-byte.
|
||||
- Observable completion: a round that regresses recall is reverted and the configuration is restored byte-for-byte, while a compliant round is accepted and persisted, proven by a self-improvement integration test.
|
||||
@@ -218,7 +218,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: improve/evolve.py_
|
||||
- _Depends: 7.3, 7.4, 4.1_
|
||||
|
||||
- [ ] 7.6 Auto-shadow precision-draggers, nominate coverage gaps, and separate the objective
|
||||
- [x] 7.6 Auto-shadow precision-draggers, nominate coverage gaps, and separate the objective
|
||||
- Auto-shadow a rule that drags aggregate precision below the gate rather than delete it, preserving recall while removing the false-positive cost.
|
||||
- Nominate a new or loosened rule through the signal file when a coverage gap surfaces as a persistent miss, leaving pattern authoring to a human pull request.
|
||||
- Treat maximizing the project score as the optimization objective and the recall/FP gate as a separate hard feasibility constraint that is never folded into the score reward.
|
||||
|
||||
Reference in New Issue
Block a user