The meta-agent round journalled `no_proposal`. The real cause was an
authentication failure: `_model_config` hard-coded the anthropic provider
while the detector was talking to DeepSeek and no ANTHROPIC_API_KEY exists
here. The composition itself was fine -- MetaAgent constructed, evolve
started, budget $5 -- and the trace was thrown away.
`no_proposal` covered two different worlds: a proposer that considered the
facts and declined, and one that could not start at all. The journal now
carries the proposer's own reason, and the vendor comes from
`[harnessx] provider` instead of being assumed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The 176-pair vfc baseline hit its four-hour timeout and wrote nothing. Scores
were accumulated in memory and persisted once at the end, so roughly four
hours of paid API calls were lost. I fixed the exception path this morning --
a provider hiccup costs one pair -- and missed that a timeout, a kill or a
machine going away cost all of them.
Each pair is now appended to `scores.jsonl` and flushed as it is measured, and
`--resume` keeps what a previous run of that exact baseline already bought.
This also gives the per-pair outcomes the round-3 review asked for, which make
the noise floor auditable after the fact instead of only in aggregate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning
candidates`, and the stop-or-go reading. No model is called anywhere in this
group, so the condition that could have ended the feature was reachable for
free.
Verdict: go.
agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection
vfc (deciding) 84.7%, baseline still running, non-gating
vfc-js (deciding) 52.9% -- see below
github (deciding) 100.0%
vibe-py (development)100.0% against 83.3%
The margin is largest on agent-vfc, which is the slice the tool is actually
for.
The measurement forced a shape correction before the detector existed.
config_secrets first read 63.6% with a bind-and-literal shape, because a
permissive default is as often an argument to a call -- cors({origin: '*'}),
app.use(session({secure: false})) -- as it is a binding. Adding call sites to
that family's shape took it to 100% and recovered four agent-vfc rows.
Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because
eight of seventeen rows label a function the JavaScript CST names `<anon>` --
`module.exports.publish = function () {}` and its relatives. That is a
function-naming gap, it caps the existing hunter on that slice exactly as
much, and it is the same class as the `_declarator_start` defects found in
learning-harness task 4.1.
The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a
measurement rather than a remembered number.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
records whether its decision beat noise
Two things the maintainer was right about.
Thinking was disabled so `temperature: 0` would not be silently ignored --
a trade that only pays if it buys determinism. Round zero says it did not:
11 of 20 injection pairs disagreed with themselves across five runs, so the
variance is in the agentic path, not the decoder. We paid the reasoning cost
on a reasoning model and got nothing. It is now `[learning] thinking`,
recorded in the artifact and part of its key so the two modes cannot be read
as one number. The default stays off only because that is what round zero
was measured under.
And the deeper point: the spec built `sign_test` and `reliable_change` and
the acceptance rule calls neither -- `decide` compares raw deltas. On a
detector with that floor a raw delta is mostly noise, and a rule that
brittle is exactly what makes a deterministic decoder look necessary. It is
not; the alternative is an acceptance rule that models the score as noisy,
the way reflective-evolution and Bayesian prompt optimizers do. Changing
Req 9.5 is a maintainer's call, so this does not change the decision: every
round now journals the two-sided sign test over the pairs that moved, and
after four rounds the journal will say whether any decision could be told
from a coin flip.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Keyed by model alone, the memory-safety run over the vfc slice silently
replaced the run over the web families, and the published table would have
shown one of them as if it were everything. The artifact key and the
comparison label now carry the slice selection when it is not the default.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The chat client meters itself and nothing passed the meter to the baseline
or the round, so every published cost would have been 0.00 -- and the
round's cost cap, which reads the same meter, could never have fired.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
`--slice vfc` selected the vfc rows and `run_baseline` filtered them all out
again against its own default of the two web slices, so the memory-safety
baseline Req 8.4 asks for produced an empty report and said nothing about it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of nine group-3 tasks were rejected. Each defect was hiding a number,
and the numbers are why the loop was not learning:
- The fixed side was judged against the *vulnerable* side's function
ranges. 191 of 227 pairs with a resolvable labeled span (84%) have
different spans on the two sides, so a leak was recorded as silence,
always in the direction that flatters Youden.
- Round zero measured every family with the generalist. `family:unknown`
earns no credit against any real family, so every family but `unknown`
baselined at a structural zero and every noise floor with it.
- The cross-family sweep counted absolute failures instead of negative
flips. On a corpus where most pairs fail at round zero that rejects
every round for damage it did not do. Unscorable rows voted too.
- K runs was declared and never threaded; `pairs --k-runs` was parsed and
discarded, and its only test grepped `--help`.
- The meta-agent proposer was a stub, so `learning round` could never
propose anything. It now composes MetaAgent.evolve against a scratch
copy of the family directory, and `--proposals FILE` runs a recorded
sequence offline.
- `refused_holdout` was a vocabulary word no code path could produce, and
the covering test accepted either outcome.
- Cost was metered once, after everything had been paid for; a stage that
raised left no record at all.
- The corrected lever number was published beside the earlier one rather
than in place of it, and the test named for that property never checked.
- Refusals were `continue` statements: `Refusal.degradation()` existed and
nothing called it. A train-split identical twin was reported as
`split:train`, which is untrue.
Test repairs that each changed a number: two split tests measured an
`x = 1` fixture that teaches nothing whatever the rule says; two CLI
scoreboard tests asserted against all-zero output produced by a stub that
answers twice and puts its finding outside the labeled function.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.9. Five entry points make the classified detector set reachable:
classify a corpus and report the classifier against itself, score a catalog by
family, run round zero, run one evolve round, and publish every number from its
artifacts. The pairs command gains a runs-per-pair flag, since the family scorer
needs more than one.
Every step degrades with a reason rather than failing: with no chat endpoint the
run says so and still records what it could measure without a detector, and a
missing configuration directory is written from the current hunter prompt rather
than being an error. Artifacts are keyed by model throughout, so comparing a
second detector is a flag rather than a rewrite.
Both matrices are green and all three gate outputs stay byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.4. A standard scan now classifies each hotspot region and runs the
detectors of every admitted family plus the generalist, reading the
configurations round zero wrote. A scan never creates them: a missing directory
is a recorded reason and the stage is skipped, which is also what happens when
no chat endpoint resolves.
Every claim passes through its family's verifier and carries what produced it:
the family, the detector version and what the verifier said. Nothing rises above
suspicion here, because the verifier writes a tag and the evidence ladder
belongs to another boundary. Reports gain a section naming the three, SARIF
gains the matching properties, and the manifest gains a learning block with the
taxonomy version, the per-family counts, the classifier report and the round.
The feature is off unless a project turns it on, so an existing scan is
unchanged, and all three gate outputs stay byte-identical.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1. The obligations proposer runs after variant search in standard and
deep scans, reads the declared policy from the target, and never writes it.
Its findings are appended after every other finding and never enter the sandbox
candidate set. The manifest block carries the sibling sets, the under-populated
count, the operations examined, the findings by label, the policy version, the
stage degradations, and the sibling sets themselves.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Review rounds 3-5 remediations (Req 5, 7).
- ousast improve resolves candidates from the target's calibration dir,
then the working directory's (where `mechanisms export` writes), and
admits into <target>/.openultrasast/calibration/mechanisms.jsonl,
where a scan of that target reads; --mechanism-candidates,
--mechanism-store, --no-mechanisms; --dry-run redirects the store to
scratch; accepted rounds print admit/retract <id>.
- run_improvement threads the store paths; RoundOutcome carries the
mechanism edits; reverted mechanism keys are blocked outright; a
no-gain admission is rejected as mechanism_no_gain; a retraction may
name an admitted record that left the candidate set.
- leave-one-out forwards variant-search degradations and pins the
containment and label rules; mechanisms export defaults to the
candidates file so it never re-admits a tombstone.
- design §Lever rewritten to the implemented gate; README and docs show
export -> improve -> scan.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 5.1 (Req 6.1, 6.3).
- reports.py: the markdown and SARIF writers take the mechanisms a run
cites; a finding cites one through its mechanism:<id> tag (variant) or
its merged overlay record's mechanism_id; markdown shows Mechanism,
Known fix guard and Learned from pairs per finding plus a Mechanisms
section, SARIF properties carry mechanism_id/summary/guard/pairs;
variant findings keep suspicion, merged inventory findings keep their
own evidence level; reports without mechanisms are unchanged.
- cli: _search_variants returns the cited mechanisms from the store and
the scan passes them to both writers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 3.1 (Req 4.1-4.4).
- semantic/loo.py: every scorable pair is held out once; a temporary
store is seeded from the other seeded/reviewed pairs (advisory and
title pairs are scored as targets, never teach; known_limit skipped);
both excerpts are searched for variants and scored with the pair rules
(detected inside the labeled function naming the labeled mechanism or
sink; silent when the fixed side yields no hit); found_by and leaked_by
name the records and the pairs that taught them; metrics per slice,
profile and mechanism; score_pair_with_store is shared with the lever.
- cli: `pairs --slice S --loo [--loo-out P]` writes loo.json; never a gate.
- First measurement: vibe-py 35 pairs, 11 detected, 21 silent, Youden
-0.086, 16 teaching pairs.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 2.3 (Req 3.4, 3.5, 3.6, 6.2).
- [variants] enabled / max_mechanisms config; MAP runs variant search
after the overlay in standard and deep scans only, appends the variant
findings, rewrites overlay.json with merged ids, and the manifest
records mechanisms_searched, files_searched, findings and
merged_into_overlay.
- REGRESS: variant findings join the candidate list after the promotions
as zero-score hotspots, under the existing safety check and cap.
- Spec notes for group 2 and the design synced (variant_search.py,
seed.py, derive_shape signature).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.3 (Req 10.1-10.5).
- pairs.py: catalog rows with vendored = false carry their harvest recipe
and no excerpts; evaluate_catalog harvests them into
OPENULTRASAST_PAIR_CACHE (default ~/.cache/openultrasast/pairs) only
when `pairs --pointers` or OPENULTRASAST_PAIRS_NETWORK=1 allows the
network, otherwise records pointer_pair_skipped (a failed fetch is
pointer_pair_fetch_failed with detail); a warm cache never scores
without the switch. select_vendored for tests and gates.
- harvest.py: materialize_pointer harvests into the cache without the
license requirement (nothing is redistributed); redaction applies.
- catalog_gen.py: emits pointer rows with recipe fields; training
manifests skip them.
- pair_gate and improve/evolve.evaluate_profiles score vendored pairs
only; cli `pairs --pointers`; .gitignore covers the cache path.
- tests: loader, skip/degradation, cache hit, fetch failure, warm cache
with network off, profile gate exclusion, harvest and generator paths;
conftest keeps every test offline whatever the shell exports.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 6.3.
- improve/evolve.py: evaluate_profiles and profile_regressions score the
holdout pair split per provenance profile every round; a drop in pair
pass rate or Youden beyond --profile-tolerance rejects the round with
reason profile_regression:<profile>; profiles under
--min-holdout-pairs are reported, not gated; journal records both.
- cli improve: --pair-catalog, --no-pair-gate, --profile-tolerance,
--min-holdout-pairs; a missing catalog fails loud instead of skipping
the gate.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 6.1.
- provenance.py: classify a checkout as human / agent / mixed / unknown
from deterministic signals only (.claude/, AGENTS.md, Lovable meta,
agent git trailers on a bounded commit sample); no model involved.
- cli scan runs a `provenance` stage; the manifest records the result.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 1.2, 2.1-2.5 and 6.2 (pair-side filters).
- pairs.py: coverage with a source counts as detection and as leak;
function-scoped matching by line-range containment from the vulnerable
side's parse (unresolvable labels are counted loss, never silent
misses); exact or alias sink matching; weak CWE-only labels never match;
side-by-side inventory/overlay/hunter scorers; per_profile,
per_mechanism, achievable, known_limit and loss payload; loader rejects
unknown mechanism/provenance/split and reserved or missing function
labels on harvested slices; candidate rulesets thread through for the
improve loop.
- overlay.py: OverlayRecord.function (additive) filled from the shared
naming helper in one parse; benchmark.py: ExpectedFinding.mechanism.
- cli pairs: --slice over every slice, --profile, --split, --hunter,
--hunter-model.
- catalogs migrated to the new schema (provenance, split, function,
mechanism, known_limit for owasp-java-hash); baseline recorded in the
spec notes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Stdlib parser only; pytest skips it so tests cannot pick up a developer key.
The embedding model is openai/text-embedding-3-small via OpenRouter /embeddings.
Regex inventory stays the proposer. MAP attaches promote/demote/unadjudicated/coverage
from intra-file taint (tree-sitter when a grammar parses, Python ast as last resort).
Deep sandboxes promotions only, ordered by the existing heuristic ranker plus optional
OpenRouter mechanism recall. The SAST pair slice scores overlay outcomes; the stage-1
detection gate and local pair gate stay inventory-only.
A zero-dependency MCP server over stdio (newline-delimited JSON-RPC 2.0) for
OpenCode/IDE integration, exposing only stable project-level operations.
- mcp.py: McpServer (pure handle() dispatch) + serve() stdio loop, stdlib-only.
Ten tools: openultrasast.scan/status/findings/get_finding/evidence/artifacts/
benchmark/explain/propose_patch/export_report. scan/benchmark run the bounded
analysis pipeline; NO tool accepts a free-form command — no shell, Docker, or
internal hunter tools are exposed. propose_patch degrades visibly (patch oracle
is a later phase). initialize/tools/list/tools/call/ping handled; notifications
get no response; errors surface as JSON-RPC errors instead of crashing the loop.
- cli: `ousast mcp` subcommand launches the stdio server (lazy import → keeps the
mcp↔cli cycle one-directional).
- tests/test_mcp.py: tool listing is exactly the narrow surface, surface-safety
(no shell/exec/command tool), scan→status→findings→explain→evidence flow,
export_report, propose_patch degradation, and the stdio loop round-trip.
- README "MCP server" section; openrouter-sast-harness Phase 12 checked off.
Verified end-to-end over real stdio (initialize → tools/list → scan). 179 passed,
4 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a [harnessx] config block so the agentic plane can be steered from
openultrasast.toml, and make the LLM provider selectable (the hunter and verifier
previously hardcoded AnthropicProvider).
- config.HarnessxConfig: provider ("anthropic"|"openai"|"litellm", default
anthropic), max_cost_usd (2.0), token_threshold (120000); parsed from [harnessx].
- harness_ext.build_provider: single lazy, capability-guarded provider factory.
The provider name is validated before any import, so an unknown choice fails with
a clear ValueError even when the extra is absent; each provider reads its own
standard key env var (openai -> OPENAI_API_KEY, anthropic -> ANTHROPIC_API_KEY).
- HxScanOrchestrator and verify_findings_dispatch take a provider and route through
build_provider; cli threads config.harnessx.{provider,max_cost_usd,token_threshold}
into both the hunter and the verifier.
- README: "Configuring the HarnessX agentic plane" section (install, models,
provider, budgets, fallback/degradation).
Test isolation: convert the "harnessx not in sys.modules" import-guards to a
hermetic subprocess fixture (tests/conftest.py assert_cold_of_harnessx) and
simulate extra-absent in the CLI dispatch tests, so the suite is deterministic
whether or not the optional extra is installed (it now is, locally). Verified the
OpenAI path constructs end-to-end (build_provider -> OpenAIProvider; orchestrator
-> Harness) with the budget threaded through.
137 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Expose the bounded, gated self-improvement loop as a first-class subcommand so it
can be steered from the command line / opencode.
- `ousast improve <manifest>` resolves the target from the benchmark manifest,
runs the loop to convergence, and writes accepted edits to the loop-owned ledger
at <target>/.openultrasast/calibration/rule_policy.json — exactly where `scan`
and `benchmark` read it, so a subsequent scan of that target automatically
picks up the improved ruleset (the loop closes end-to-end).
- Flags: --max-rounds, --recall-floor, --fp-ceiling, --ledger, --journal,
--ruleset-dir, and --dry-run (runs against a throwaway ledger, touches nothing).
- Per-round output: reason, recall/fp/score deltas, and the edits applied.
- tests/test_cli_improve.py: accept path writes the loop-owned ledger (noisy rule
auto-shadowed), --dry-run never touches it, missing manifest fails loudly.
- README: quickstart entry + a "Rule-level loop: ousast improve" subsection
documenting the bounded levers, hard acceptance gate, and the HarnessX MetaAgent
as the optional proposer over the same gated machinery.
131 passed, 3 skipped; ruff + mypy clean; detection gate PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adopt HarnessX at the hunter loop and verifier, capability-gated so the default
zero-dependency scan is byte-identical and CI (no extra) exercises the fallback.
- 6.1/6.2 build_sast() (single, lazy HarnessX composition site in harness_ext.py,
mirrors DocuHarnessX make_docgen) + HxScanOrchestrator (hunter_harness.py): a
drop-in for run_hunter_pool that runs one bounded HarnessX agent task per ranked
target (ModelConfig(main=AnthropicProvider).agentic(build_sast()).run(BaseTask)),
parses findings from a JSON structured-output contract, returns the existing
HunterPoolResult, maps scan_id->session_id and emits per-target exit_reason/cost/
steps to the outer trace.
- 6.4 verify_judge.verify_findings_dispatch: a drop-in superset of
verification.verify_findings — structural verifier verbatim by default (rejects
unknown reachability), HarnessX llm-judge (build_judge_prompt/parse_judge_response)
when the extra + a verifier model are present; verdict mapped to VerificationResult
evidence-gated (plausible>=0.6 accepted; unsupported/refused rejected; infra
failure never auto-rejects).
- 6.5 cli capability gate (hx_hunter/hx_verify): selects HxScanOrchestrator /
judge only in standard mode with a configured model AND has_harnessx(); otherwise
falls back and records a {stage,reason:harnessx_extra_unavailable} degradation in
the manifest (new append-only write_manifest degradations kwarg). With no model
configured the manifest stays byte-identical.
- pyproject: mypy override ignore_missing_imports for harnessx.* (absent in the
type-check env).
Verified the HarnessX path constructs against the real extra in a throwaway venv
(build_sast -> HarnessConfig; orchestrator -> Harness; build_judge_prompt/
parse_judge_response/verdict mapping) — only the live agent.run()/provider.complete()
LLM calls are offline-untestable and stay gated. 109 tests pass (2 construction
smokes skip without the extra), ruff + mypy clean, detection gate 93.6%/0% intact.
Tasks 6.1/6.2/6.4/6.5 done; 6.3 (deterministic stages as HarnessX processors) deferred.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Complete Phase 2: make benchmark output drive per-rule improvement, wire the
loop-owned ledger, and turn a confirmed false positive into a score knob. Zero
new dependencies; detection stays byte-identical (recall 44/47 = 93.6%, FP 0%)
and the 90/10 gate holds.
- Per-rule attribution (4.1): BenchmarkMiss/BenchmarkFalsePositive carry rule_id;
BenchmarkMetrics.per_rule aggregates matched/missed/false_positives per rule.
- Recommendation delta (4.2): evaluate_benchmark emits RuleRecommendation
(keep/shadow/loosen) written to rule_policy_delta.json; the inert
next_improvement_candidate stub is now rule-aware ("rule:<id>" or
"language_hunter").
- Loop ledger (4.3): cli loads the ruleset with the rule_policy.json overlay
(empty -> byte-identical; an entry overlays a rule's status/floors at load);
read_rule_ledger/write_rule_ledger added for the evolve loop. A disabled rule
in the ledger no longer fires (covered by an integration test).
- FP -> reachability knob (4.4): a confirmed false-positive learning lowers a
finding's effective reachability multiplier in the score
(fp_reachability_overrides + scoring reachability_override) instead of deleting
the rule, so recall stays high while the score stops over-penalizing.
93 tests pass; ruff + mypy clean. Spec tasks 4.1-4.4 checked off — Phase 2
(Core: rules-as-data + benchmark signals) is complete. The loop-driven decision
to apply ledger flips remains the gated evolve loop (Phase 3/4).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Externalize detection rules to versioned TOML and remove rule-local severity, so
rules detect and the central CWE policy decides severity — fixing "rules too
influential / not centrally managed". Zero-dependency; detection is byte-identical
(benchmark recall 44/47 = 93.6%, FP 0% unchanged) and the 90/10 gate holds.
- RulesetStore (ruleset/store.py + ruleset/<lang>/rules.toml): PatternRule as data
(rule_id, title, languages, cwe, tags, pattern, status, min_evidence_level,
precision_estimate, version) with NO severity; load_ruleset (sorted, with
optional rule_policy.json ledger overlay) and a stdlib TOML write_ruleset. The
32 rules were generated from the prior PATTERN_RULES with a verified exact
round-trip (pattern/cwe/languages/tags/title).
- findings.py: drop the hardcoded PatternRule class + PATTERN_RULES tuple; load the
default ruleset from TOML; quick_scan_findings(root, targets, rankings, ruleset,
policy) resolves each finding's severity from policy via resolve_severity
(SEVERITY_LABEL maps 0-5 -> string) and tags findings with their rule status.
- Three-status volume control: StaticFinding gains status (default "enabled");
disabled rules never fire; shadow findings are partitioned out of the report and
score in the CLI (written to shadow_findings.json) — no shadow rules by default.
- Emission floors (RulesetConfig.min_emit_priority / min_emit_precision, default
0.0 = emit all) replace "every match is a finding".
- CLI loads the ruleset as a stage and uses it for the fail-loud policy check,
detection, hunter pool, and the score CWE map. Wheel ships the ruleset TOML.
85 tests pass; ruff + mypy clean. Spec tasks 3.1-3.4 checked off.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the HarnessX self-improving loop without an agent in the loop. Every scan
now feeds its verifier outcomes back into the next scan's ranking:
- New `calibrate` stage (after rank, before findings) loads a persistent ledger
of scoped false-positive learnings and demotes every previously-rejected scope
(`calibrate_rankings`), writing `calibration/applied_calibrations.json`.
- New `record_calibration` stage (after verify) converts this run's non-accepted
verifier outcomes into scoped `FalsePositiveLearning` records
(`learnings_from_verifications`) and merges them into
`.openultrasast/calibration/false_positive_learnings.json`, de-duplicated by
finding ID so a repeat rejection does not compound without bound.
- Lower the calibration floor (MIN_CALIBRATED_PRIORITY = 0.1) so accumulated
rejections can demote a scope below the natural ranking minimum.
Demotion is scoped and reversible: a rejection in one path never touches an
accepted finding in another, and priority is demoted rather than the finding
deleted so the audit trail explains every downgrade. First run is a no-op
(empty ledger), so existing single-run behavior is unchanged.
Adds tests/test_pipeline_calibration.py (two-run demotion, order-independent),
documents the realized loop in the README, and checks off the
standard_security_harness milestone in tasks.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Rebuild the quick-scan detection engine and ground-truth benchmarks to hit
>=90% recall / <10% false positives for C/C++, Python, JavaScript, and Java.
Engine (findings.py, preprocess.py):
- Scope each PatternRule by language (fixes cross-language false positives such
as the Python eval/exec rule firing on JS .exec()).
- Add JavaScript/TypeScript rules (command exec, eval/Function, reflected XSS,
SQLi, path traversal, SSRF, weak hash, deserialization); execFile/spawn-argv
excluded as safe alternatives.
- Add Java rules (Runtime exec, SQL string concatenation, weak hash, readObject)
and a Groovy server-side template rule (yieldUnescaped of a bound variable).
- Broaden C/C++ and Python coverage; precision guards for format strings and
parameterised vs concatenated SQL; #-comment skipping; CWE in rationale.
- Register .java/.groovy/.tpl source extensions.
Benchmarks:
- Add realistic corpora modeled on Damn Vulnerable C/C++, PyGoat/DVPWA, and
NodeGoat, plus the real kiview/damn-vulnerable-spring-boot-app vendored
verbatim (MIT, attribution in LICENSE/NOTICE.md). Each manifest carries
complete, independently-labeled ground truth including genuinely
hard-to-detect vulns (integer overflow, second-order SQLi, prototype
pollution) so recall is measured honestly, plus safe-API files for precision.
- Fix pre-existing smoke-manifest ground-truth errors (Python/JS line offsets,
missing C/C++ strcpy entry, Java SQLi line).
benchmark.py:
- Fix sink matching to normalise the expected sink consistently so sinks with
underscores (e.g. render_template_string) can match.
Tests:
- Add tests/test_detection_benchmarks.py enforcing the >=90% recall / <10% FP
goal per language and overall.
Measured: C/C++ 93.3%, Python 93.3%, JavaScript 92.3%, Java 100%;
overall 93.6% recall, 0.0% false-positive rate.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
