mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 19:59:26 +00:00
feat(benchmark): Phase 2 (task 4) — actionable per-rule signals, ledger, FP knob
Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.
- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
(keep/shadow/loosen) written to rule_policy_delta.json; the inert
next_improvement_candidate stub is now rule-aware ("rule:<id>" or
"language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
(empty -> byte-identical; an entry overlays a rule's status/floors at load);
read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
finding's effective reachability multiplier in the score
(fp_reachability_overrides + scoring reachability_override) instead of deleting
the rule, so recall stays high while the score stops over-penalizing.
93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
444d8c9490
commit
2d7ef48e52
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "harnessx-self-improving-rulesets",
|
||||
"created_at": "2026-06-22T10:07:18Z",
|
||||
"updated_at": "2026-06-22T10:45:00Z",
|
||||
"updated_at": "2026-06-22T13:45:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -90,22 +90,22 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: PolicyScoringProcessor, RulesetStore, PolicyStore_
|
||||
- _Depends: 3.1, 3.2, 1.2_
|
||||
|
||||
- [ ] 4. Make benchmark output an actionable per-rule signal
|
||||
- [ ] 4.1 Attribute misses and false positives to rules
|
||||
- [x] 4. Make benchmark output an actionable per-rule signal
|
||||
- [x] 4.1 Attribute misses and false positives to rules
|
||||
- Attribute every recorded miss and every recorded false positive to a rule identifier and aggregate per-rule precision and recall in the benchmark metrics.
|
||||
- Observable completion: a benchmark run reports per-rule precision/recall and every miss and false positive carries a rule identifier, verified by a metrics test.
|
||||
- _Requirements: 4.1_
|
||||
- _Boundary: Benchmark Harness_
|
||||
- _Depends: 3.1_
|
||||
|
||||
- [ ] 4.2 Emit a per-rule recommendation delta artifact
|
||||
- [x] 4.2 Emit a per-rule recommendation delta artifact
|
||||
- Replace the inert improvement-candidate stub with a per-rule recommendation artifact recommending enable, disable, shadow, threshold, or evidence-floor changes per rule.
|
||||
- Observable completion: a benchmark run writes a recommendation artifact with per-rule suggested actions alongside its records, verified against the documented delta schema.
|
||||
- _Requirements: 4.2_
|
||||
- _Boundary: Benchmark Harness_
|
||||
- _Depends: 4.1_
|
||||
|
||||
- [ ] 4.3 Introduce the loop-owned ledger and re-key calibration to rules
|
||||
- [x] 4.3 Introduce the loop-owned ledger and re-key calibration to rules
|
||||
- Add the loop-owned overlay ledger that records per-rule status, evidence floor, threshold, and precision, applied over the ruleset at load time.
|
||||
- Re-key calibration from the first-tag key to the rule identifier and write status flips into the ledger rather than only demoting ranking.
|
||||
- Observable completion: an empty ledger leaves behavior byte-identical to the prior phase while a ledger entry overlays a rule's status at load, proven by an integration test.
|
||||
@@ -113,7 +113,7 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: RulesetStore, Calibration subsystem_
|
||||
- _Depends: 3.1, 3.2_
|
||||
|
||||
- [ ] 4.4 Convert false-positive confirmation into a reachability knob
|
||||
- [x] 4.4 Convert false-positive confirmation into a reachability knob
|
||||
- When confirming a false positive, lower the finding's effective reachability multiplier rather than delete the rule, so recall stays high while the score stops over-penalizing.
|
||||
- Observable completion: confirming a false positive lowers the finding's effective reachability multiplier and the rule still fires, proven by a calibration test.
|
||||
- _Requirements: 5.6_
|
||||
|
||||
Reference in New Issue
Block a user