Review rounds 3-5 remediations (Req 5, 7).
- ousast improve resolves candidates from the target's calibration dir,
then the working directory's (where `mechanisms export` writes), and
admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
where a scan of that target reads; --mechanism-candidates,
--mechanism-store, --no-mechanisms; --dry-run redirects the store to
scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
mechanism edits; reverted mechanism keys are blocked outright; a
no-gain admission is rejected as mechanism_no_gain; a retraction may
name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
containment and label rules; mechanisms export defaults to the
candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
export -> improve -> scan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.1 (Req 6.1, 6.3).
- reports.py: the markdown and SARIF writers take the mechanisms a run
cites; a finding cites one through its mechanism:<id> tag (variant) or
its merged overlay record's mechanism_id; markdown shows Mechanism,
Known fix guard and Learned from pairs per finding plus a Mechanisms
section, SARIF properties carry mechanism_id/summary/guard/pairs;
variant findings keep suspicion, merged inventory findings keep their
own evidence level; reports without mechanisms are unchanged.
- cli: _search_variants returns the cited mechanisms from the store and
the scan passes them to both writers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.2 (Req 5.3, 5.4, 5.5).
- propose_mechanism_edits: retract admitted records that leak on a
holdout fixed side; admit a candidate only when a trial store holding
it alone recovers a currently missed holdout pair without leaking.
- run_round(mechanism_candidates=, mechanism_store=): validates, applies
after the rule gate passed, re-scores the holdout pairs per profile with
variant search (evaluate_mechanism_profiles), and rejects with a
byte-for-byte revert on mechanism_leak, profile_regression or
mechanism_no_gain; the journal records each mechanism edit with its
pair provenance and the metrics before and after.
- The rule gate and its per-profile clause are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.1 (Req 5.1, 5.2, 5.5).
- validator: MechanismEdit (admit | retract by record id) under the
`mechanisms` lever; validate_mechanism accepts only records from the
exporter candidate set with origin corpus, a closed guard and source
kinds, and identifier-only shape text; free-form shapes are rejected.
- apply_mechanism_edits appends the record (admit) or a retracted
tombstone (retract) so the log stays append-only; StoreSnapshot.restore
reverts byte for byte. MechanismStore.load folds tombstones away.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1 (Req 4.1-4.4).
- semantic/loo.py: every scorable pair is held out once; a temporary
store is seeded from the other seeded/reviewed pairs (advisory and
title pairs are scored as targets, never teach; known_limit skipped);
both excerpts are searched for variants and scored with the pair rules
(detected inside the labeled function naming the labeled mechanism or
sink; silent when the fixed side yields no hit); found_by and leaked_by
name the records and the pairs that taught them; metrics per slice,
profile and mechanism; score_pair_with_store is shared with the lever.
- cli: `pairs --slice S --loo [--loo-out P]` writes loo.json; never a gate.
- First measurement: vibe-py 35 pairs, 11 detected, 21 silent, Youden
-0.086, 16 teaching pairs.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- variant_search.search_tree skips a malformed store row with a
variants_store_row_invalid degradation instead of aborting the scan;
a hit on a demoted or flowless overlay record is never merged (test).
- seed.pair_lessons is the one shape derivation shared by the exporter
and leave-one-out; labels pass line and sink together; pairs without a
labeled function are reported as no_labeled_function.
- config parses [variants].max_mechanisms with the shared int helper.
- regress.candidate.hotspot_from_finding is the one zero-score hotspot
helper for forced inventory and variant findings.
- design synced (SearchResult, hits_to_findings summaries, candidate.py).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3 (Req 3.4, 3.5, 3.6, 6.2).
- [variants] enabled / max_mechanisms config; MAP runs variant search
after the overlay in standard and deep scans only, appends the variant
findings, rewrites overlay.json with merged ids, and the manifest
records mechanisms_searched, files_searched, findings and
merged_into_overlay.
- REGRESS: variant findings join the candidate list after the promotions
as zero-score hotspots, under the existing safety check and cap.
- Spec notes for group 2 and the design synced (variant_search.py,
seed.py, derive_shape signature).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.2 (Req 3.1-3.3, 3.6).
- variants.match_shapes: call sites whose trailing callee name and arity
match a shape and whose source positions all carry a source of any
kind; constants and arity mismatches never match.
- semantic/variant_search.py: search_tree over a tree's parsed files with
the store's corpus shapes (bounded by max_mechanisms, language
degradations recorded); hits_to_findings merges a hit into an overlay
promote/coverage record at the same call site (mechanism_id, no
duplicate) and otherwise emits variant:<id>:<path>:<line> at suspicion
with the pair provenance in the rationale.
- OverlayRecord.mechanism_id (additive).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.1 (Req 1.1, 1.2, 1.5, 2.5).
- semantic/seed.py: export_mechanisms seeds only seeded/reviewed,
non-known-limit, parsing pairs with cached excerpts; every skip carries
a reason (tier, known_limit, parse_failed, language unsupported with a
degradation, pointer not cached, no labeled sink call site); runs
offline over the vendored catalog.
- variants.derive_shape: labels that name only a function and a mechanism
resolve to the fact-sink call inside that function (CWE match and a
non-constant argument first); line and sink intersect when both exist;
container reads are subscript or .get( only.
- append_from_pair keeps the strongest review tier and accumulates
provenance tags when a shape is re-seeded.
- CLI: `ousast mechanisms export --slice S [--store P] [--json]`.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.2 (Req 1.3, 1.4).
- Mechanism gains origin, review_tier, pairs, shape, guard with defaults;
rows written before this spec load unchanged.
- MechanismStore.load folds the append-only log by id (latest row wins)
so a re-seeded shape carries every pair that taught it.
- append_from_pair writes origin=corpus rows with a deterministic id
derived from the shape key; corpus_mechanisms selects searchable rows.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.1 (Req 2.1-2.4). Spec approved by the maintainer in session.
- semantic/variants.py: Shape (language, sink trailing name, arity,
source positions and kinds, closed guard kind, mechanism id) with a
stable text-free key; derive_shape finds the labeled sink call site in
the vulnerable FileIR, traces argument identifiers through binds to a
fact source, a parameter or a container read, and classifies the guard
the fixed side added from the labeled function's added statements
against a closed set (none when unknown). No path, line or literal
enters a shape; a constant-only call teaches nothing.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.4 (Req 10.6, 5.1).
- harvest.merge_recipes appends recipes by name and never rewrites
existing rows.
- vibe-py/build_recipes.py --pointers: the 40 LLM-generated repositories
without a license become vendored = false rows (80 pairs, agent,
seeded) appended to recipes.toml; unlicensed human repos stay skipped.
- agent-vfc/build_recipes.py --from ... --pointers [--cache]: unlicensed
agent-authored fixes become title-tier pointer rows whose function
label is derived from the cached excerpt (30 added from a fresh
662-candidate search); TOML booleans rendered correctly.
- catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer),
agent-vfc 59 (29 + 30); no excerpt is vendored for any pointer row.
- `pairs --slice vibe-py --pointers` scores both provenances (human 35,
agent 80); the default run skips 80 pairs with pointer_pair_skipped;
suite and gates unchanged. README/docs name the nightly command.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.3 (Req 10.1-10.5).
- pairs.py: catalog rows with vendored = false carry their harvest recipe
and no excerpts; evaluate_catalog harvests them into
OPENULTRASAST_PAIR_CACHE (default ~/.cache/openultrasast/pairs) only
when `pairs --pointers` or OPENULTRASAST_PAIRS_NETWORK=1 allows the
network, otherwise records pointer_pair_skipped (a failed fetch is
pointer_pair_fetch_failed with detail); a warm cache never scores
without the switch. select_vendored for tests and gates.
- harvest.py: materialize_pointer harvests into the cache without the
license requirement (nothing is redistributed); redaction applies.
- catalog_gen.py: emits pointer rows with recipe fields; training
manifests skip them.
- pair_gate and improve/evolve.evaluate_profiles score vendored pairs
only; cli `pairs --pointers`; .gitignore covers the cache path.
- tests: loader, skip/degradation, cache hit, fetch failure, warm cache
with network off, profile gate exclusion, harvest and generator paths;
conftest keeps every test offline whatever the shell exports.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.2 (Req 9.3).
- improve/evolve.py: evaluate_profiles filters holdout pairs to
GATING_TIERS before computing per-profile metrics, so advisory and
title pairs are scored by `pairs` but never gate a round; a regression
on reviewed pairs still rejects with profile_regression:<profile>.
- tests: a regression confined to title pairs is accepted; the same
regression on reviewed pairs is rejected and the title pairs are
excluded from the gated profile rather than outvoted.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 8.1, 7.1 (tests) and the re-scoped 5.3.
- pairs.py: review_tier in {seeded, advisory, title, reviewed} with
per-slice defaults (local reviewed, vibe-py seeded, sast/vfc/vfc-js/
github advisory, agent-vfc title); reviewer = "pending" means title, a
named reviewer means reviewed, reviewed without a reviewer is rejected;
per_tier metrics in the payload; select_tier and GATING_TIERS for the
improve loop.
- catalog_gen.py: stamps review_tier (and reviewer) on every generated
row and training manifest row; title rows enter catalog.toml; the
catalog-candidates.toml queue is retired; precedence matches the loader.
- agent-vfc: 29 title-tier rows load and score (0/29, Youden 0.0, the
honest unreviewed baseline); local rows name their reviewer.
- Tests: tier defaults, loader rejections, per_tier payload, the Req 8.1
catalog-on-disk / stage-1 exclusion proof and per-slice CLI checks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 6.3.
- improve/evolve.py: evaluate_profiles and profile_regressions score the
holdout pair split per provenance profile every round; a drop in pair
pass rate or Youden beyond --profile-tolerance rejects the round with
reason profile_regression:<profile>; profiles under
--min-holdout-pairs are reported, not gated; journal records both.
- cli improve: --pair-catalog, --no-pair-gate, --profile-tolerance,
--min-holdout-pairs; a missing catalog fails loud instead of skipping
the gate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 6.1.
- provenance.py: classify a checkout as human / agent / mixed / unknown
from deterministic signals only (.claude/, AGENTS.md, Lovable meta,
agent git trailers on a bounded commit sample); no model involved.
- cli scan runs a `provenance` stage; the manifest records the result.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 1.2, 2.1-2.5 and 6.2 (pair-side filters).
- pairs.py: coverage with a source counts as detection and as leak;
function-scoped matching by line-range containment from the vulnerable
side's parse (unresolvable labels are counted loss, never silent
misses); exact or alias sink matching; weak CWE-only labels never match;
side-by-side inventory/overlay/hunter scorers; per_profile,
per_mechanism, achievable, known_limit and loss payload; loader rejects
unknown mechanism/provenance/split and reserved or missing function
labels on harvested slices; candidate rulesets thread through for the
improve loop.
- overlay.py: OverlayRecord.function (additive) filled from the shared
naming helper in one parse; benchmark.py: ExpectedFinding.mechanism.
- cli pairs: --slice over every slice, --profile, --split, --hunter,
--hunter-model.
- catalogs migrated to the new schema (provenance, split, function,
mechanism, known_limit for owasp-java-hash); baseline recorded in the
spec notes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 3.1, 3.2 (harvest library) and the Scorer-owned naming helper.
- semantic/functions.py: one source of function names for scorer, overlay
and catalog generator (declarator wins over parser name; JS/TS bare-param
arrows and undeclared methods are <anon>; C++ Class::method resolves to
its unqualified tail; Perl sub; statements never name).
- benchmarks/pairs/harvest.py: shared library with name / line_range /
hunk / enclosing modes, comment-blind masking (triple-quote aware),
upstream_start headers, code-safe quoted-only redaction.
- benchmarks/pairs/catalog_gen.py: derive function labels and sinks from
the declarator at the mapped sink location, CWE-family sink tokens,
--prune with refusal when nothing derived, fails loud without the
package; vfc/harvest.py and vfc/generate_catalog.py become shims.
- tests/test_pair_harvest.py replaces test_vfc_harvest.py.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Add openultrasast[semantic] with official grammar wheels and one CST walker
into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI
stays extra-free; a semantic job runs pytest -m semantic.
Stdlib parser only; pytest skips it so tests cannot pick up a developer key.
The embedding model is openai/text-embedding-3-small via OpenRouter /embeddings.
Regex inventory stays the proposer. MAP attaches promote/demote/unadjudicated/coverage
from intra-file taint (tree-sitter when a grammar parses, Python ast as last resort).
Deep sandboxes promotions only, ordered by the existing heuristic ranker plus optional
OpenRouter mechanism recall. The SAST pair slice scores overlay outcomes; the stage-1
detection gate and local pair gate stay inventory-only.
Replace the hardcoded skill-name stub with a real, routable skill index.
- skills.py (zero-dep): SkillDescriptor inventory (~22 ToB skills with routing
metadata — languages, tags, vulnerability classes, stages — plus a short operating
checklist each), aligned with the spec's routing examples. route_skills scores by
vulnerability-class/tag/stage signal with a language filter+bonus, and never selects
on a pure language match. route_skills_for_target derives vuln classes from tags;
select_skill_context packs checklists into a char budget; skill_chunks chunks the
guidance into the `skills` retrieval namespace.
- hunter.select_skill_snippets now delegates to the router (existing C-memory and
crypto+auth routing expectations preserved).
- hunter_harness._describe feeds the budget-capped routed skill context into the
HarnessX hunter prompt (the genuine consumer of skill_budget_chars).
- 8.4 zero-dep guard extended to cover skills.py.
tests/test_skills.py (9 tests: C parser → memory/fuzzing, crypto → constant-time/
vector, language-mismatch + language-only exclusion, stage filtering, budget cap,
namespace chunking). openrouter-sast-harness Phase 14 checked off.
188 passed, 4 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
