mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(ruleset): Phase 2 (task 3) — rules as governed data, severity from policy
Externalize detection rules to versioned TOML and remove rule-local severity, so rules detect and the central CWE policy decides severity — fixing "rules too influential / not centrally managed". Zero-dependency; detection is byte-identical (benchmark recall 44/47 = 93.6%, FP 0% unchanged) and the 90/10 gate holds. - RulesetStore (ruleset/store.py + ruleset/<lang>/rules.toml): PatternRule as data (rule_id, title, languages, cwe, tags, pattern, status, min_evidence_level, precision_estimate, version) with NO severity; load_ruleset (sorted, with optional rule_policy.json ledger overlay) and a stdlib TOML write_ruleset. The 32 rules were generated from the prior PATTERN_RULES with a verified exact round-trip (pattern/cwe/languages/tags/title). - findings.py: drop the hardcoded PatternRule class + PATTERN_RULES tuple; load the default ruleset from TOML; quick_scan_findings(root, targets, rankings, ruleset, policy) resolves each finding's severity from policy via resolve_severity (SEVERITY_LABEL maps 0-5 -> string) and tags findings with their rule status. - Three-status volume control: StaticFinding gains status (default "enabled"); disabled rules never fire; shadow findings are partitioned out of the report and score in the CLI (written to shadow_findings.json) — no shadow rules by default. - Emission floors (RulesetConfig.min_emit_priority / min_emit_precision, default 0.0 = emit all) replace "every match is a finding". - CLI loads the ruleset as a stage and uses it for the fail-loud policy check, detection, hunter pool, and the score CWE map. Wheel ships the ruleset TOML. 85 tests pass; ruff + mypy clean. Spec tasks 3.1-3.4 checked off. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
fb7594a3ed
commit
444d8c9490
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "harnessx-self-improving-rulesets",
|
||||
"created_at": "2026-06-22T10:07:18Z",
|
||||
"updated_at": "2026-06-22T10:20:00Z",
|
||||
"updated_at": "2026-06-22T10:45:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -59,15 +59,15 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
|
||||
## Phase 2 — Core: Rules-as-Data and Benchmark Signals (zero-dep, byte-identical)
|
||||
|
||||
- [ ] 3. Convert detection rules into versioned, governed data
|
||||
- [ ] 3.1 Externalize detection patterns into a versioned Ruleset Store
|
||||
- [x] 3. Convert detection rules into versioned, governed data
|
||||
- [x] 3.1 Externalize detection patterns into a versioned Ruleset Store
|
||||
- Load detection rules from versioned data files in which each rule declares a pattern, a CWE, tags, and a loop-controlled status, carrying no rule-local severity field.
|
||||
- Remove the rule-local severity field from the in-memory rule model so severity can only come from policy.
|
||||
- Observable completion: the existing pattern rules load from versioned data files with no severity attribute present, proven by a unit test asserting the absence of the field.
|
||||
- _Requirements: 2.1, 2.9_
|
||||
- _Boundary: RulesetStore_
|
||||
|
||||
- [ ] 3.2 Implement the three-status volume control
|
||||
- [x] 3.2 Implement the three-status volume control
|
||||
- Support exactly three rule statuses — `enabled`, `shadow`, `disabled` — where enabled fires/scores/reports, shadow fires and records outcomes but is excluded from report and score, and disabled does not fire.
|
||||
- Exclude shadow-status findings from the reported set and from the score input while still recording their outcome for precision tracking.
|
||||
- Observable completion: an enabled rule appears in report and score, a shadow rule is excluded from both yet records an outcome, and a disabled rule never fires, proven by a unit test across all three statuses.
|
||||
@@ -75,14 +75,14 @@ This plan splits the fused regex layer into three governed owners (rules detect,
|
||||
- _Boundary: RulesetStore, ScoreModel_
|
||||
- _Depends: 3.1, 2.1_
|
||||
|
||||
- [ ] 3.3 Remap the unmapped C/C++ buffer rules to a scored CWE
|
||||
- [x] 3.3 Remap the unmapped C/C++ buffer rules to a scored CWE
|
||||
- Remap the C/C++ buffer-handling rules that reference the unmapped CWE-120 to CWE-121 so they resolve in policy and contribute a nonzero penalty.
|
||||
- Observable completion: the remapped buffer rules resolve to a severity-5 policy entry and produce a nonzero score penalty, verified by a unit test.
|
||||
- _Requirements: 2.8_
|
||||
- _Boundary: RulesetStore_
|
||||
- _Depends: 3.1, 1.1_
|
||||
|
||||
- [ ] 3.4 Wire findings and severity resolution through the governance plane
|
||||
- [x] 3.4 Wire findings and severity resolution through the governance plane
|
||||
- Pass the loaded ruleset into the finding-detection stage instead of reading an implicit module-level rule list, and resolve every finding's severity through the Policy Store.
|
||||
- Apply emission thresholds so not every regex match becomes a reported finding, replacing the old "rule is its own volume control" behavior.
|
||||
- Observable completion: a scan produces findings with policy-resolved severity and shadow findings filtered out, while the default ruleset keeps benchmark output byte-identical to the pre-feature baseline.
|
||||
|
||||
Reference in New Issue
Block a user