feat(benchmark): Phase 2 (task 4) — actionable per-rule signals, ledger, FP knob

Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.

- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
  BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
  (keep/shadow/loosen) written to rule_policy_delta.json; the inert
  next_improvement_candidate stub is now rule-aware ("rule:<id>" or
  "language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
  (empty -> byte-identical; an entry overlays a rule's status/floors at load);
  read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
  in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
  finding's effective reachability multiplier in the score
  (fp_reachability_overrides + scoring reachability_override) instead of deleting
  the rule, so recall stays high while the score stops over-penalizing.

93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
Marius Ciepluch
2026-06-22 15:41:17 +02:00
co-authored by Claude Opus 4.8
parent 444d8c9490
commit 2d7ef48e52
11 changed files with 316 additions and 16 deletions
@@ -1,7 +1,7 @@
{
"feature_name": "harnessx-self-improving-rulesets",
"created_at": "2026-06-22T10:07:18Z",
"updated_at": "2026-06-22T10:45:00Z",
"updated_at": "2026-06-22T13:45:00Z",
"language": "en",
"phase": "implementation",
"approvals": {
@@ -90,22 +90,22 @@ This plan splits the fused regex layer into three governed owners (rules detect,
- _Boundary: PolicyScoringProcessor, RulesetStore, PolicyStore_
- _Depends: 3.1, 3.2, 1.2_
- [ ] 4. Make benchmark output an actionable per-rule signal
- [ ] 4.1 Attribute misses and false positives to rules
- [x] 4. Make benchmark output an actionable per-rule signal
- [x] 4.1 Attribute misses and false positives to rules
- Attribute every recorded miss and every recorded false positive to a rule identifier and aggregate per-rule precision and recall in the benchmark metrics.
- Observable completion: a benchmark run reports per-rule precision/recall and every miss and false positive carries a rule identifier, verified by a metrics test.
- _Requirements: 4.1_
- _Boundary: Benchmark Harness_
- _Depends: 3.1_
- [ ] 4.2 Emit a per-rule recommendation delta artifact
- [x] 4.2 Emit a per-rule recommendation delta artifact
- Replace the inert improvement-candidate stub with a per-rule recommendation artifact recommending enable, disable, shadow, threshold, or evidence-floor changes per rule.
- Observable completion: a benchmark run writes a recommendation artifact with per-rule suggested actions alongside its records, verified against the documented delta schema.
- _Requirements: 4.2_
- _Boundary: Benchmark Harness_
- _Depends: 4.1_
- [ ] 4.3 Introduce the loop-owned ledger and re-key calibration to rules
- [x] 4.3 Introduce the loop-owned ledger and re-key calibration to rules
- Add the loop-owned overlay ledger that records per-rule status, evidence floor, threshold, and precision, applied over the ruleset at load time.
- Re-key calibration from the first-tag key to the rule identifier and write status flips into the ledger rather than only demoting ranking.
- Observable completion: an empty ledger leaves behavior byte-identical to the prior phase while a ledger entry overlays a rule's status at load, proven by an integration test.
@@ -113,7 +113,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
- _Boundary: RulesetStore, Calibration subsystem_
- _Depends: 3.1, 3.2_
- [ ] 4.4 Convert false-positive confirmation into a reachability knob
- [x] 4.4 Convert false-positive confirmation into a reachability knob
- When confirming a false positive, lower the finding's effective reachability multiplier rather than delete the rule, so recall stays high while the score stops over-penalizing.
- Observable completion: confirming a false positive lowers the finding's effective reachability multiplier and the rule still fires, proven by a calibration test.
- _Requirements: 5.6_