mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
feat(benchmark): Phase 2 (task 4) — actionable per-rule signals, ledger, FP knob
Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.
- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
(keep/shadow/loosen) written to rule_policy_delta.json; the inert
next_improvement_candidate stub is now rule-aware ("rule:<id>" or
"language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
(empty -> byte-identical; an entry overlays a rule's status/floors at load);
read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
finding's effective reachability multiplier in the score
(fp_reachability_overrides + scoring reachability_override) instead of deleting
the rule, so recall stays high while the score stops over-penalizing.
93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
444d8c9490
commit
2d7ef48e52
@@ -5,6 +5,8 @@ from .store import (
|
||||
PatternRule,
|
||||
RulesetError,
|
||||
load_ruleset,
|
||||
read_rule_ledger,
|
||||
write_rule_ledger,
|
||||
write_ruleset,
|
||||
)
|
||||
|
||||
@@ -13,5 +15,7 @@ __all__ = [
|
||||
"PatternRule",
|
||||
"RulesetError",
|
||||
"load_ruleset",
|
||||
"read_rule_ledger",
|
||||
"write_rule_ledger",
|
||||
"write_ruleset",
|
||||
]
|
||||
|
||||
@@ -81,6 +81,17 @@ def write_ruleset(path: Path, rules: Iterable[PatternRule]) -> None:
|
||||
path.write_text("\n\n".join(blocks) + "\n")
|
||||
|
||||
|
||||
def read_rule_ledger(path: Path) -> dict[str, dict[str, object]]:
|
||||
"""Read the loop-owned ``rule_policy.json`` overlay (status/floors per rule_id)."""
|
||||
return _load_ledger(path)
|
||||
|
||||
|
||||
def write_rule_ledger(path: Path, entries: dict[str, dict[str, object]]) -> None:
|
||||
"""Write the loop-owned ``rule_policy.json`` overlay (used by the evolve loop)."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(entries, indent=2, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
def _rule_from_dict(item: dict[str, object], path: Path) -> PatternRule:
|
||||
try:
|
||||
rule_id = str(item["rule_id"])
|
||||
|
||||
Reference in New Issue
Block a user