Task 3.1 (Req 4.1-4.4).
- semantic/loo.py: every scorable pair is held out once; a temporary
store is seeded from the other seeded/reviewed pairs (advisory and
title pairs are scored as targets, never teach; known_limit skipped);
both excerpts are searched for variants and scored with the pair rules
(detected inside the labeled function naming the labeled mechanism or
sink; silent when the fixed side yields no hit); found_by and leaked_by
name the records and the pairs that taught them; metrics per slice,
profile and mechanism; score_pair_with_store is shared with the lever.
- cli: `pairs --slice S --loo [--loo-out P]` writes loo.json; never a gate.
- First measurement: vibe-py 35 pairs, 11 detected, 21 silent, Youden
-0.086, 16 teaching pairs.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3 (Req 3.4, 3.5, 3.6, 6.2).
- [variants] enabled / max_mechanisms config; MAP runs variant search
after the overlay in standard and deep scans only, appends the variant
findings, rewrites overlay.json with merged ids, and the manifest
records mechanisms_searched, files_searched, findings and
merged_into_overlay.
- REGRESS: variant findings join the candidate list after the promotions
as zero-score hotspots, under the existing safety check and cap.
- Spec notes for group 2 and the design synced (variant_search.py,
seed.py, derive_shape signature).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.3 (Req 10.1-10.5).
- pairs.py: catalog rows with vendored = false carry their harvest recipe
and no excerpts; evaluate_catalog harvests them into
OPENULTRASAST_PAIR_CACHE (default ~/.cache/openultrasast/pairs) only
when `pairs --pointers` or OPENULTRASAST_PAIRS_NETWORK=1 allows the
network, otherwise records pointer_pair_skipped (a failed fetch is
pointer_pair_fetch_failed with detail); a warm cache never scores
without the switch. select_vendored for tests and gates.
- harvest.py: materialize_pointer harvests into the cache without the
license requirement (nothing is redistributed); redaction applies.
- catalog_gen.py: emits pointer rows with recipe fields; training
manifests skip them.
- pair_gate and improve/evolve.evaluate_profiles score vendored pairs
only; cli `pairs --pointers`; .gitignore covers the cache path.
- tests: loader, skip/degradation, cache hit, fetch failure, warm cache
with network off, profile gate exclusion, harvest and generator paths;
conftest keeps every test offline whatever the shell exports.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 6.3.
- improve/evolve.py: evaluate_profiles and profile_regressions score the
holdout pair split per provenance profile every round; a drop in pair
pass rate or Youden beyond --profile-tolerance rejects the round with
reason profile_regression:<profile>; profiles under
--min-holdout-pairs are reported, not gated; journal records both.
- cli improve: --pair-catalog, --no-pair-gate, --profile-tolerance,
--min-holdout-pairs; a missing catalog fails loud instead of skipping
the gate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 6.1.
- provenance.py: classify a checkout as human / agent / mixed / unknown
from deterministic signals only (.claude/, AGENTS.md, Lovable meta,
agent git trailers on a bounded commit sample); no model involved.
- cli scan runs a `provenance` stage; the manifest records the result.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 1.2, 2.1-2.5 and 6.2 (pair-side filters).
- pairs.py: coverage with a source counts as detection and as leak;
function-scoped matching by line-range containment from the vulnerable
side's parse (unresolvable labels are counted loss, never silent
misses); exact or alias sink matching; weak CWE-only labels never match;
side-by-side inventory/overlay/hunter scorers; per_profile,
per_mechanism, achievable, known_limit and loss payload; loader rejects
unknown mechanism/provenance/split and reserved or missing function
labels on harvested slices; candidate rulesets thread through for the
improve loop.
- overlay.py: OverlayRecord.function (additive) filled from the shared
naming helper in one parse; benchmark.py: ExpectedFinding.mechanism.
- cli pairs: --slice over every slice, --profile, --split, --hunter,
--hunter-model.
- catalogs migrated to the new schema (provenance, split, function,
mechanism, known_limit for owasp-java-hash); baseline recorded in the
spec notes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Stdlib parser only; pytest skips it so tests cannot pick up a developer key.
The embedding model is openai/text-embedding-3-small via OpenRouter /embeddings.
Regex inventory stays the proposer. MAP attaches promote/demote/unadjudicated/coverage
from intra-file taint (tree-sitter when a grammar parses, Python ast as last resort).
Deep sandboxes promotions only, ordered by the existing heuristic ranker plus optional
OpenRouter mechanism recall. The SAST pair slice scores overlay outcomes; the stage-1
detection gate and local pair gate stay inventory-only.
A zero-dependency MCP server over stdio (newline-delimited JSON-RPC 2.0) for
OpenCode/IDE integration, exposing only stable project-level operations.
- mcp.py: McpServer (pure handle() dispatch) + serve() stdio loop, stdlib-only.
Ten tools: openultrasast.scan/status/findings/get_finding/evidence/artifacts/
benchmark/explain/propose_patch/export_report. scan/benchmark run the bounded
analysis pipeline; NO tool accepts a free-form command — no shell, Docker, or
internal hunter tools are exposed. propose_patch degrades visibly (patch oracle
is a later phase). initialize/tools/list/tools/call/ping handled; notifications
get no response; errors surface as JSON-RPC errors instead of crashing the loop.
- cli: `ousast mcp` subcommand launches the stdio server (lazy import → keeps the
mcp↔cli cycle one-directional).
- tests/test_mcp.py: tool listing is exactly the narrow surface, surface-safety
(no shell/exec/command tool), scan→status→findings→explain→evidence flow,
export_report, propose_patch degradation, and the stdio loop round-trip.
- README "MCP server" section; openrouter-sast-harness Phase 12 checked off.
Verified end-to-end over real stdio (initialize → tools/list → scan). 179 passed,
4 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).
- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
The provider name is validated before any import, so an unknown choice fails with
a clear ValueError even when the extra is absent; each provider reads its own
standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
provider, budgets, fallback/degradation).
Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.
137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expose the bounded, gated self-improvement loop as a first-class subcommand so it
can be steered from the command line / opencode.
- `ousast improve <manifest>` resolves the target from the benchmark manifest,
runs the loop to convergence, and writes accepted edits to the loop-owned ledger
at <target>/.openultrasast/calibration/rule_policy.json — exactly where `scan`
and `benchmark` read it, so a subsequent scan of that target automatically
picks up the improved ruleset (the loop closes end-to-end).
- Flags: --max-rounds, --recall-floor, --fp-ceiling, --ledger, --journal,
--ruleset-dir, and --dry-run (runs against a throwaway ledger, touches nothing).
- Per-round output: reason, recall/fp/score deltas, and the edits applied.
- tests/test_cli_improve.py: accept path writes the loop-owned ledger (noisy rule
auto-shadowed), --dry-run never touches it, missing manifest fails loudly.
- README: quickstart entry + a "Rule-level loop: ousast improve" subsection
documenting the bounded levers, hard acceptance gate, and the HarnessX MetaAgent
as the optional proposer over the same gated machinery.
131 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adopt HarnessX at the hunter loop and verifier, capability-gated so the default
zero-dependency scan is byte-identical and CI (no extra) exercises the fallback.
- 6.1/6.2 build_sast() (single, lazy HarnessX composition site in harness_ext.py,
mirrors DocuHarnessX make_docgen) + HxScanOrchestrator (hunter_harness.py): a
drop-in for run_hunter_pool that runs one bounded HarnessX agent task per ranked
target (ModelConfig(main=AnthropicProvider).agentic(build_sast()).run(BaseTask)),
parses findings from a JSON structured-output contract, returns the existing
HunterPoolResult, maps scan_id->session_id and emits per-target exit_reason/cost/
steps to the outer trace.
- 6.4 verify_judge.verify_findings_dispatch: a drop-in superset of
verification.verify_findings — structural verifier verbatim by default (rejects
unknown reachability), HarnessX llm-judge (build_judge_prompt/parse_judge_response)
when the extra + a verifier model are present; verdict mapped to VerificationResult
evidence-gated (plausible>=0.6 accepted; unsupported/refused rejected; infra
failure never auto-rejects).
- 6.5 cli capability gate (hx_hunter/hx_verify): selects HxScanOrchestrator /
judge only in standard mode with a configured model AND has_harnessx(); otherwise
falls back and records a {stage,reason:harnessx_extra_unavailable} degradation in
the manifest (new append-only write_manifest degradations kwarg). With no model
configured the manifest stays byte-identical.
- pyproject: mypy override ignore_missing_imports for harnessx.* (absent in the
type-check env).
Verified the HarnessX path constructs against the real extra in a throwaway venv
(build_sast -> HarnessConfig; orchestrator -> Harness; build_judge_prompt/
parse_judge_response/verdict mapping) — only the live agent.run()/provider.complete()
LLM calls are offline-untestable and stay gated. 109 tests pass (2 construction
smokes skip without the extra), ruff + mypy clean, detection gate 93.6%/0% intact.
Tasks 6.1/6.2/6.4/6.5 done; 6.3 (deterministic stages as HarnessX processors) deferred.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.
- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
(keep/shadow/loosen) written to rule_policy_delta.json; the inert
next_improvement_candidate stub is now rule-aware ("rule:<id>" or
"language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
(empty -> byte-identical; an entry overlays a rule's status/floors at load);
read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
finding's effective reachability multiplier in the score
(fp_reachability_overrides + scoring reachability_override) instead of deleting
the rule, so recall stays high while the score stops over-penalizing.
93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Externalize detection rules to versioned TOML and remove rule-local severity, so
rules detect and the central CWE policy decides severity — fixing "rules too
influential / not centrally managed". Zero-dependency; detection is byte-identical
(benchmark recall 44/47 = 93.6%, FP 0% unchanged) and the 90/10 gate holds.
- RulesetStore (ruleset/store.py + ruleset/<lang>/rules.toml): PatternRule as data
(rule_id, title, languages, cwe, tags, pattern, status, min_evidence_level,
precision_estimate, version) with NO severity; load_ruleset (sorted, with
optional rule_policy.json ledger overlay) and a stdlib TOML write_ruleset. The
32 rules were generated from the prior PATTERN_RULES with a verified exact
round-trip (pattern/cwe/languages/tags/title).
- findings.py: drop the hardcoded PatternRule class + PATTERN_RULES tuple; load the
default ruleset from TOML; quick_scan_findings(root, targets, rankings, ruleset,
policy) resolves each finding's severity from policy via resolve_severity
(SEVERITY_LABEL maps 0-5 -> string) and tags findings with their rule status.
- Three-status volume control: StaticFinding gains status (default "enabled");
disabled rules never fire; shadow findings are partitioned out of the report and
score in the CLI (written to shadow_findings.json) — no shadow rules by default.
- Emission floors (RulesetConfig.min_emit_priority / min_emit_precision, default
0.0 = emit all) replace "every match is a finding".
- CLI loads the ruleset as a stage and uses it for the fail-loud policy check,
detection, hunter pool, and the score CWE map. Wheel ships the ruleset TOML.
85 tests pass; ruff + mypy clean. Spec tasks 3.1-3.4 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:
- New `calibrate` stage (after rank, before findings) loads a persistent ledger
of scoped false-positive learnings and demotes every previously-rejected scope
(`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into
`.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
rejections can demote a scope below the natural ranking minimum.
Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.
Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Rebuild the quick-scan detection engine and ground-truth benchmarks to hit
>=90% recall / <10% false positives for C/C++, Python, JavaScript, and Java.
Engine (findings.py, preprocess.py):
- Scope each PatternRule by language (fixes cross-language false positives such
as the Python eval/exec rule firing on JS .exec()).
- Add JavaScript/TypeScript rules (command exec, eval/Function, reflected XSS,
SQLi, path traversal, SSRF, weak hash, deserialization); execFile/spawn-argv
excluded as safe alternatives.
- Add Java rules (Runtime exec, SQL string concatenation, weak hash, readObject)
and a Groovy server-side template rule (yieldUnescaped of a bound variable).
- Broaden C/C++ and Python coverage; precision guards for format strings and
parameterised vs concatenated SQL; #-comment skipping; CWE in rationale.
- Register .java/.groovy/.tpl source extensions.
Benchmarks:
- Add realistic corpora modeled on Damn Vulnerable C/C++, PyGoat/DVPWA, and
NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored
verbatim (MIT, attribution in LICENSE/NOTICE.md). Each manifest carries
complete, independently-labeled ground truth including genuinely
hard-to-detect vulns (integer overflow, second-order SQLi, prototype
pollution) so recall is measured honestly, plus safe-API files for precision.
- Fix pre-existing smoke-manifest ground-truth errors (Python/JS line offsets,
missing C/C++ strcpy entry, Java SQLi line).
benchmark.py:
- Fix sink matching to normalise the expected sink consistently so sinks with
underscores (e.g. render_template_string) can match.
Tests:
- Add tests/test_detection_benchmarks.py enforcing the >=90% recall / <10% FP
goal per language and overall.
Measured: C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0.0% false-positive rate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
