- tasks.md: all tasks checked; rounds 1-6 recorded with RED evidence,
measured leave-one-out and admission numbers, and the recorded design
deviations.
- spec.json: completed 2026-09-05.
- roadmap: corpus-seeded-mechanisms marked completed; the loop is closed
end to end (export -> improve admits under the holdout clause -> scan
searches -> reports name the mechanism).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Review rounds 3-5 remediations (Req 5, 7).
- ousast improve resolves candidates from the target's calibration dir,
then the working directory's (where `mechanisms export` writes), and
admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
where a scan of that target reads; --mechanism-candidates,
--mechanism-store, --no-mechanisms; --dry-run redirects the store to
scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
mechanism edits; reverted mechanism keys are blocked outright; a
no-gain admission is rejected as mechanism_no_gain; a retraction may
name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
containment and label rules; mechanisms export defaults to the
candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
export -> improve -> scan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.2 (Req 5.3, 5.4, 5.5).
- propose_mechanism_edits: retract admitted records that leak on a
holdout fixed side; admit a candidate only when a trial store holding
it alone recovers a currently missed holdout pair without leaking.
- run_round(mechanism_candidates=, mechanism_store=): validates, applies
after the rule gate passed, re-scores the holdout pairs per profile with
variant search (evaluate_mechanism_profiles), and rejects with a
byte-for-byte revert on mechanism_leak, profile_regression or
mechanism_no_gain; the journal records each mechanism edit with its
pair provenance and the metrics before and after.
- The rule gate and its per-profile clause are unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- variant_search.search_tree skips a malformed store row with a
variants_store_row_invalid degradation instead of aborting the scan;
a hit on a demoted or flowless overlay record is never merged (test).
- seed.pair_lessons is the one shape derivation shared by the exporter
and leave-one-out; labels pass line and sink together; pairs without a
labeled function are reported as no_labeled_function.
- config parses [variants].max_mechanisms with the shared int helper.
- regress.candidate.hotspot_from_finding is the one zero-score hotspot
helper for forced inventory and variant findings.
- design synced (SearchResult, hits_to_findings summaries, candidate.py).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3 (Req 3.4, 3.5, 3.6, 6.2).
- [variants] enabled / max_mechanisms config; MAP runs variant search
after the overlay in standard and deep scans only, appends the variant
findings, rewrites overlay.json with merged ids, and the manifest
records mechanisms_searched, files_searched, findings and
merged_into_overlay.
- REGRESS: variant findings join the candidate list after the promotions
as zero-score hotspots, under the existing safety check and cap.
- Spec notes for group 2 and the design synced (variant_search.py,
seed.py, derive_shape signature).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 1.1 (Req 2.1-2.4). Spec approved by the maintainer in session.
- semantic/variants.py: Shape (language, sink trailing name, arity,
source positions and kinds, closed guard kind, mechanism id) with a
stable text-free key; derive_shape finds the labeled sink call site in
the vulnerable FileIR, traces argument identifiers through binds to a
fact source, a parameter or a container read, and classifies the guard
the fixed side added from the labeled function's added statements
against a closed set (none when unknown). No path, line or literal
enters a shape; a constant-only call teaches nothing.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: round-12 verdict and the decision to assemble the redaction
test's example key at runtime and rewrite the unpushed series so no
commit carries a contiguous key literal; two dated notes marked as
superseded by tasks 8.1 and 8.4.
- roadmap: the folded inventory-vs-overlay Youden item is checked.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
- tasks.md: all tasks checked; the two escalations are closed (5.1 by the
Req 10 pointer rows, 5.3 by the Req 9 review tiers); round 11 recorded
with the carry-forwards for corpus-seeded-mechanisms and
overlay-ir-completeness.
- roadmap: measured state after tiers and pointer slices (vibe-py 115,
agent-vfc 59, gates byte-identical to pre-spec); pair-corpus-honesty
marked completed.
- spec.json: completed 2026-09-05 after eleven review rounds.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.4 (Req 10.6, 5.1).
- harvest.merge_recipes appends recipes by name and never rewrites
existing rows.
- vibe-py/build_recipes.py --pointers: the 40 LLM-generated repositories
without a license become vendored = false rows (80 pairs, agent,
seeded) appended to recipes.toml; unlicensed human repos stay skipped.
- agent-vfc/build_recipes.py --from ... --pointers [--cache]: unlicensed
agent-authored fixes become title-tier pointer rows whose function
label is derived from the cached excerpt (30 added from a fresh
662-candidate search); TOML booleans rendered correctly.
- catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer),
agent-vfc 59 (29 + 30); no excerpt is vendored for any pointer row.
- `pairs --slice vibe-py --pointers` scores both provenances (human 35,
agent 80); the default run skips 80 pairs with pointer_pair_skipped;
suite and gates unchanged. README/docs name the nightly command.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.3 (Req 10.1-10.5).
- pairs.py: catalog rows with vendored = false carry their harvest recipe
and no excerpts; evaluate_catalog harvests them into
OPENULTRASAST_PAIR_CACHE (default ~/.cache/openultrasast/pairs) only
when `pairs --pointers` or OPENULTRASAST_PAIRS_NETWORK=1 allows the
network, otherwise records pointer_pair_skipped (a failed fetch is
pointer_pair_fetch_failed with detail); a warm cache never scores
without the switch. select_vendored for tests and gates.
- harvest.py: materialize_pointer harvests into the cache without the
license requirement (nothing is redistributed); redaction applies.
- catalog_gen.py: emits pointer rows with recipe fields; training
manifests skip them.
- pair_gate and improve/evolve.evaluate_profiles score vendored pairs
only; cli `pairs --pointers`; .gitignore covers the cache path.
- tests: loader, skip/degradation, cache hit, fetch failure, warm cache
with network off, profile gate exclusion, harvest and generator paths;
conftest keeps every test offline whatever the shell exports.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Two engine specs drafted after the pair-corpus baseline, awaiting owner
approval before implementation:
- corpus-seeded-mechanisms: structural shapes derived from trusted pairs,
export, variant search merged with overlay flows, leave-one-out
measurement, and a closed `mechanisms` lever for the improve loop.
- reachability-flow-model: call graph over FunctionIR, reachability from
entry points to sinks, hunter client preferring HarnessX (task 4.1b).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Add openultrasast[semantic] with official grammar wheels and one CST walker
into FileIR. quick stays extra-free. Host CLI is not overlay-ready. Default CI
stays extra-free; a semantic job runs pytest -m semantic.
Regex inventory stays the proposer. MAP attaches promote/demote/unadjudicated/coverage
from intra-file taint (tree-sitter when a grammar parses, Python ast as last resort).
Deep sandboxes promotions only, ordered by the existing heuristic ranker plus optional
OpenRouter mechanism recall. The SAST pair slice scores overlay outcomes; the stage-1
detection gate and local pair gate stay inventory-only.
Replace the hardcoded skill-name stub with a real, routable skill index.
- skills.py (zero-dep): SkillDescriptor inventory (~22 ToB skills with routing
metadata — languages, tags, vulnerability classes, stages — plus a short operating
checklist each), aligned with the spec's routing examples. route_skills scores by
vulnerability-class/tag/stage signal with a language filter+bonus, and never selects
on a pure language match. route_skills_for_target derives vuln classes from tags;
select_skill_context packs checklists into a char budget; skill_chunks chunks the
guidance into the `skills` retrieval namespace.
- hunter.select_skill_snippets now delegates to the router (existing C-memory and
crypto+auth routing expectations preserved).
- hunter_harness._describe feeds the budget-capped routed skill context into the
HarnessX hunter prompt (the genuine consumer of skill_budget_chars).
- 8.4 zero-dep guard extended to cover skills.py.
tests/test_skills.py (9 tests: C parser → memory/fuzzing, crypto → constant-time/
vector, language-mismatch + language-only exclusion, stage filtering, budget cap,
namespace chunking). openrouter-sast-harness Phase 14 checked off.
188 passed, 4 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A zero-dependency MCP server over stdio (newline-delimited JSON-RPC 2.0) for
OpenCode/IDE integration, exposing only stable project-level operations.
- mcp.py: McpServer (pure handle() dispatch) + serve() stdio loop, stdlib-only.
Ten tools: openultrasast.scan/status/findings/get_finding/evidence/artifacts/
benchmark/explain/propose_patch/export_report. scan/benchmark run the bounded
analysis pipeline; NO tool accepts a free-form command — no shell, Docker, or
internal hunter tools are exposed. propose_patch degrades visibly (patch oracle
is a later phase). initialize/tools/list/tools/call/ping handled; notifications
get no response; errors surface as JSON-RPC errors instead of crashing the loop.
- cli: `ousast mcp` subcommand launches the stdio server (lazy import → keeps the
mcp↔cli cycle one-directional).
- tests/test_mcp.py: tool listing is exactly the narrow surface, surface-safety
(no shell/exec/command tool), scan→status→findings→explain→evidence flow,
export_report, propose_patch degradation, and the stdio loop round-trip.
- README "MCP server" section; openrouter-sast-harness Phase 12 checked off.
Verified end-to-end over real stdio (initialize → tools/list → scan). 179 passed,
4 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Express the deterministic scan stages as model-disabled processors carrying work in
named slots under a read/write allow-list validated on every lifecycle hook.
- slot_contract.py (zero-dep): SlotContractError, SlotContractMixin (reads_slots/
writes_slots/skip_model=True, validates reads pre-hook and writes post-hook),
SlotPipeline (runs stages in order, enforces the model-disabled invariant, emits
stage_start/stage_end lifecycle hooks). Preserves the ProcessorSpec.reads/writes
provenance discipline without forcing deterministic stages through HarnessX's agent
loop ("bridge, do not map").
- stage_processors.py: Preprocess/EntryPoint/Rank/QuickFindings/PolicyScoring/Score as
skip_model slot processors; run_quick_pipeline is byte-identical to the quick path.
host_under_harnessx is the lazy, capability-gated bridge that disables the model on
HarnessX's BeforeModelEvent.skip_model (proven against the real API when the extra
is present).
- 8.4 zero-dep guard hardened: now hermetic (subprocess) and covers the two new
modules; the module-level import scan flags only column-0 imports so the intended
lazy import inside host_under_harnessx is allowed.
The sync HarnessRuntime stays the top-level driver (design "stays through Phase 4");
these slot processors are the model-disabled representation, ready for Phase 5.
tests/test_slot_contract.py (8 tests). 153 passed, 3 skipped; ruff + mypy clean;
detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
9.1 — end-to-end fallback parity: a requested-but-unavailable HarnessX standard
scan (models set, extra absent) produces byte-identical findings to the plain
deterministic path, differing only by the recorded hunter_pool/verify degradations;
quick mode is unaffected by the extra (identical findings present-or-absent, never a
degradation).
9.2 — per-slot serialization round-trip: StaticFinding / RankingScore / FileTarget /
VerificationResult / ScoreArtifact round-trip through JSON without loss, plus the real
findings.json reload path preserves nested slots (reachability evidence/conditions,
tags) rather than restoring them empty. Covers the current artifact-persistence model;
the HarnessX processor slot-contract variant stays deferred with task 6.3.
tests/test_scan_pipeline_integrity.py (8 tests). 145 passed, 3 skipped; ruff + mypy
clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Make the >=90% recall / <10% FP detection gate a first-class, score-independent
CI step with non-regression guards.
- openultrasast.gate: canonical corpus map + DetectionMetrics/DetectionGateVerdict,
evaluate_corpus/detection_gate (no score input — cannot be unblocked by score),
merge_gate (ANDs the hard detection gate with the advisory-first score gate),
within_tolerance, and a `python -m openultrasast.gate` entrypoint that exits
non-zero on a breach. Wired as an explicit step in both the GitHub and Dagger
pipelines.
- 8.1: pipeline tests prove a high/passing score cannot unblock a recall-breaching
change; the gate fails below 90% recall or at/above 10% FP.
- 8.2: tests/golden/detection_baseline.json + a byte-identical baseline test for
the default ruleset / empty ledger (44/47 = 93.62% recall, 0% FP).
- 8.3: within_tolerance band logic + a capability-gated HarnessX tolerance test.
- 8.4: zero-dependency guard — governance/scoring/benchmark planes never import
HarnessX (sys.modules check + module-scope import scan).
- test_detection_benchmarks now sources the corpus/aggregation from the gate module.
128 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stand up the deterministic self-improvement loop over the ruleset/policy
governance data. Zero-dependency and fully tested offline; detection gate intact.
- 7.1/7.2 bounded levers + validator (improve/validator.py): two levers (rule,
policy); typed edits make pattern-text and 0-5 severity changes structurally
impossible; EvolveValidator rejects bad status, unknown rule_ids (loop cannot add
rules), enabled->disabled jumps (shadow staging mandatory), unresolved CWEs, and
out-of-bounds K/MIN_SCORE, raising StrictValidationError.
- 7.3 novelty + replay + journal (improve/journal.py): a JSON round journal records
every edit/lever/outcome; reverted edits are blocked from re-proposal (novelty);
a replay smoke gate boots the candidate ruleset before re-benchmarking.
- 7.4 benchmark deltas -> rule_signals.json (recall side: miss; precision side: fp).
- 7.5 hard acceptance gate + byte-for-byte revert (improve/evolve.py): a round is
accepted only if recall >= 90% AND FP < 10% AND project score did not regress AND
no matched-finding regression; otherwise the ledger is never written (byte-for-byte
revert). run_round/run_improvement drive rounds to convergence.
- 7.6 auto-shadow precision-draggers (enabled rules with FPs -> shadow, never
delete), nominate coverage gaps via rule_signals, and keep the project score as the
reward separate from the recall/FP feasibility gate (the loop cannot raise the
score by shadowing rules that breach the gate — proven by the revert test).
Design note: the loop's safety machinery is deterministic and benchmark-driven
(the proposer is the per-rule signal); the LLM MetaAgent.evolve is the optional
richer proposer that plugs into this same validated/gated machinery when the
HarnessX extra is present (offline-untestable). This mirrors how DocuHarnessX
realizes self-improvement.
9 new tests (validator, proposer/signals, accept, byte-for-byte revert, convergence);
118 pass total, 2 construction smokes skip without the extra; ruff + mypy clean;
detection gate 93.6%/0% intact. Tasks 7.1-7.6 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
