marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-17 03:59:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(corpus-seeded-mechanisms): the mechanisms lever runs from ousast improve

Browse Source 
Review rounds 3-5 remediations (Req 5, 7).

- ousast improve resolves candidates from the target's calibration dir,
  then the working directory's (where `mechanisms export` writes), and
  admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
  where a scan of that target reads; --mechanism-candidates,
  --mechanism-store, --no-mechanisms; --dry-run redirects the store to
  scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
  mechanism edits; reverted mechanism keys are blocked outright; a
  no-gain admission is rejected as mechanism_no_gain; a retraction may
  name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
  containment and label rules; mechanisms export defaults to the
  candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
  export -> improve -> scan.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-05 15:35:08 +02:00
co-authored by Claude Fable 5.1
parent 624024e72a
commit b3b47ad757
11 changed files with 338 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.kiro/specs/corpus-seeded-mechanisms/design.md
+6 -4
View File
@@ -97,12 +97,12 @@ src/openultrasast/
│ ├── seed.py # export_mechanisms(cases, store) -> ExportReport (maintainer command `mechanisms export`)
│ └── loo.py # evaluate_loo(cases, *, store_factory) -> LooResult; payload
├── findings.py # origin "variant" via tags/finding_id prefix "variant:" (no schema change)
├── cli.py # MAP: variant search after overlay; manifest counters; `pairs --loo`; `mechanisms export --slice`
├── cli.py # MAP: variant search after overlay; manifest counters; `pairs --loo`; `mechanisms export --slice`; `improve --mechanism-candidates/--mechanism-store/--no-mechanisms` (defaults: target calibration dir, then cwd)
├── improve/validator.py# MechanismEdit lever "mechanisms"
├── improve/evolve.py # proposer from loo misses/leaks; gate unchanged
├── improve/evolve.py # proposer from loo misses/leaks; mechanism edits decided by the per-profile clause over variant search (see §Lever)
└── reports.py # mechanism id, guard, provenance on findings
tests/
├── test_variants.py, test_loo.py, test_mechanism_export.py, test_improve.py (lever)
├── test_variants.py, test_variant_search.py, test_variant_scan.py, test_mechanism_export.py, test_mechanism_seed.py, test_loo.py, test_mechanism_lever.py, test_mechanism_loop.py, test_mechanism_reports.py
```
### Modified Files
@@ -222,7 +222,9 @@ Only `seeded | reviewed` pairs seed; every trusted pair is held out once. `known
### Lever
`MechanismEdit(lever="mechanisms", action="admit"|"retract", mechanism_id, source="loo"|"export", rationale)`. Validator: id must exist in the exporter's candidate set; shape guard in `GUARD_KINDS`; no free text fields beyond rationale. Proposer: admit candidates whose leave-one-out hit recovers a currently missed holdout pair in some profile; retract admitted records that leak on holdout fixed sides. Gate unchanged; revert restores `mechanisms.jsonl` byte for byte.
`MechanismEdit(lever="mechanisms", action="admit"|"retract", mechanism_id, source="loo"|"export", rationale)`. Validator: id must exist in the exporter's candidate set; shape guard in `GUARD_KINDS`; no free text fields beyond rationale. Proposer: admit candidates whose leave-one-out hit recovers a currently missed holdout pair in some profile; retract admitted records that leak on holdout fixed sides. Application: admit appends the exporter record, retract appends a `retracted = true` tombstone (`MechanismStore.load` folds it away; the log stays append-only); `StoreSnapshot.restore` reverts byte for byte. A retraction may also name a record admitted in the scan store but no longer a candidate.
Gate (implemented 2026-09-05, deviation from the first draft's "gate unchanged"): the rule gate and its per-profile clause are unchanged for rule edits; mechanism edits are decided by the *same* `profile_regressions` clause applied to `evaluate_mechanism_profiles` (holdout pairs scored per provenance profile by variant search with the store before and after, via `loo.score_pair_with_store`), because the overlay scorer cannot see what the store changes. Rejection reasons: `mechanism_leak:<id>` (an admission fires on a holdout fixed side, §Error Handling), `profile_regression:<profile>`, `mechanism_no_gain` (no profile gains a correct pair and nothing is retracted); every rejection restores the store byte for byte and the reverted key is blocked in later rounds regardless of rationale. The exporter writes `mechanism-candidates.jsonl`; only the lever writes the scan-time `mechanisms.jsonl`.
## Data Models