Marius CiepluchandClaude Opus 5 d2ebb58007 fix(learning-harness): a hunt that ends in prose is asked to format it ...

Measured live against DeepSeek on one pair: the model spends its step budget
on tools and then answers in a paragraph that names the sink, the argument
and the line -- which parses as nothing. A detector that found the bug and
reported zero is indistinguishable from one that found nothing, and every
family in round zero would have baselined at zero for this reason alone.

The loop now takes one more turn when the hunt ends in prose or runs out of
steps, asking the provider for JSON and handing back the reasoning the model
already produced. It formats an answer that was reached; it never asks for a
new one, and a hunt that called no tool never gets there.

Same pair, same model, before and after: recall 0.00 -> 1.00, Youden 0.00 ->
1.00 over three runs per side.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG

2026-09-06 14:18:09 +02:00

..

golden

feat(gate): Phase 4 task 8 — lock the detection gate into CI

2026-06-22 17:13:02 +02:00

conftest.py

feat(pair-corpus-honesty): score non-vendored pointer pairs through a local cache

2026-09-05 10:45:43 +02:00

test_benchmark.py

feat(three-stage-scan): add aliased-sink mutation the regex scan misses

2026-09-04 11:50:44 +02:00

test_calibration.py

ci: add quality gates and release workflows

2026-06-18 10:06:21 +02:00

test_cli_hx_dispatch.py

feat(three-stage-scan): attach tool hunter to map hotspots

2026-09-04 00:04:36 +02:00

test_cli_improve.py

feat(pair-corpus-honesty): reject improve rounds that regress any provenance profile on holdout

2026-09-05 10:45:43 +02:00

test_complexity_hints.py

feat(three-stage-scan): attach test-gap hints to hotspots

2026-09-03 20:33:48 +02:00

test_complexity_ledger.py

feat(three-stage-scan): report map and honor fail-on worth-fixing

2026-09-04 00:18:33 +02:00

test_complexity_map.py

feat(three-stage-scan): attach test-gap hints to hotspots

2026-09-03 20:33:48 +02:00

test_complexity_signals.py

feat(three-stage-scan): compute complexity signals for hotspot ranking

2026-09-03 20:05:13 +02:00

test_config.py

feat: load OpenRouter settings from a local .env

2026-09-04 14:06:18 +02:00

test_degradation_matrix.py

test(learning-harness): the gate baseline and the degradation matrix

2026-09-06 13:20:23 +02:00

test_detection_benchmarks.py

feat(gate): Phase 4 task 8 — lock the detection gate into CI

2026-06-22 17:13:02 +02:00

test_findings.py

feat(ruleset): Phase 2 (task 3) — rules as governed data, severity from policy

2026-06-22 15:27:16 +02:00

test_fixture_difficulty.py

feat(learning-harness): declared twins and a fixture-difficulty check

2026-09-06 13:09:56 +02:00

test_fusion.py

feat(fusion): Phase 13 — two-panel fusion adjudication

2026-06-22 18:57:03 +02:00

test_gate.py

feat(hardening): Phase 15 — redaction, retry/backoff, budgets, threat-model docs

2026-06-22 20:02:34 +02:00

test_hardening_pipeline.py

feat(hardening): Phase 15 — redaction, retry/backoff, budgets, threat-model docs

2026-06-22 20:02:34 +02:00

test_harness_ext.py

feat(harnessx): configurable provider + per-task budget caps

2026-06-22 17:59:44 +02:00

test_harness.py

ci: add quality gates and release workflows

2026-06-18 10:06:21 +02:00

test_hunter_curated_tools.py

fix(learning-harness): a hunt that ends in prose is asked to format it

2026-09-06 14:18:09 +02:00

test_hunter_harness.py

feat(harnessx): configurable provider + per-task budget caps

2026-06-22 17:59:44 +02:00

test_hunter_tools.py

feat(three-stage-scan): add path-clamped hunter read grep and refs

2026-09-03 21:01:57 +02:00

test_hunter.py

feat(hunter): add standard-mode scheduling

2026-06-18 10:59:25 +02:00

test_improve.py

feat(pair-corpus-honesty): score non-vendored pointer pairs through a local cache

2026-09-05 10:45:43 +02:00

test_index.py

ci: add quality gates and release workflows

2026-06-18 10:06:21 +02:00

test_learning_acceptance.py

feat(learning-harness): the acceptance rule and the directory revert

2026-09-06 11:46:44 +02:00

test_learning_aggregation.py

docs(learning-harness): amend the taxonomy name and carry the flat-taxonomy caveat

2026-09-06 10:22:33 +02:00

test_learning_canaries.py

fix(learning-harness): the three defects the group 2 review found

2026-09-06 10:32:02 +02:00

test_learning_classifier_report.py

feat(learning-harness): absence rows travel with their registration

2026-09-06 12:39:55 +02:00

test_learning_classify.py

feat(learning-harness): three-tier auto-classifier

2026-09-06 09:25:45 +02:00

test_learning_cli.py

fix(learning-harness): the run reads the meter it publishes

2026-09-06 14:12:14 +02:00

test_learning_cutoff.py

fix(learning-harness): a baseline over a named slice measures that slice

2026-09-06 14:05:23 +02:00

test_learning_detectors.py

fix(learning-harness): the three defects the group 2 review found

2026-09-06 10:32:02 +02:00

test_learning_endpoint.py

test(learning-harness): apply the group 1 review suggestions

2026-09-06 09:40:24 +02:00

test_learning_families.py

test(learning-harness): apply the group 1 review suggestions

2026-09-06 09:40:24 +02:00

test_learning_journal.py

feat(learning-harness): the round journal, the archive and the rejected buffer

2026-09-06 10:10:01 +02:00

test_learning_pairs_evidence.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_pairs_path.py

feat(learning-harness): family scoring on the pairs hunter path

2026-09-06 10:41:06 +02:00

test_learning_proposer.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_publish.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_refusals.py

fix(learning-harness): the round-4 review findings

2026-09-06 14:03:10 +02:00

test_learning_round4_evidence.py

fix(learning-harness): the round-4 review findings

2026-09-06 14:03:10 +02:00

test_learning_round_evidence.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_round_loop.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_rounds.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_scan.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_scoring.py

fix(learning-harness): the three defects the group 2 review found

2026-09-06 10:32:02 +02:00

test_learning_split.py

fix(learning-harness): the round-3 review findings, remediated RED-first

2026-09-06 13:03:06 +02:00

test_learning_verifiers.py

fix(learning-harness): the three defects the group 2 review found

2026-09-06 10:32:02 +02:00

test_loo.py

fix(learning-harness): the round-4 review findings

2026-09-06 14:03:10 +02:00

test_map_gate.py

feat(three-stage-scan): gate stage-2 maps against split-sink ranking

2026-09-04 11:53:06 +02:00

test_mapping.py

feat(authorization-obligations): name handlers registered by call or convention

2026-09-06 07:31:48 +02:00

test_mcp.py

feat(three-stage-scan): add fail-on worth-fixing

2026-09-03 19:44:04 +02:00

test_mechanism_export.py

feat(corpus-seeded-mechanisms): corpus writer and additive mechanism fields

2026-09-05 14:27:33 +02:00

test_mechanism_lever.py

feat(corpus-seeded-mechanisms): the mechanisms lever runs from ousast improve

2026-09-05 15:35:08 +02:00

test_mechanism_loop.py

feat(learning-harness): one teacher rule across every learning path

2026-09-06 10:15:44 +02:00

test_mechanism_reports.py

feat(corpus-seeded-mechanisms): reports name the mechanism, its known fix guard and pair provenance

2026-09-05 15:35:08 +02:00

test_mechanism_seed.py

feat(corpus-seeded-mechanisms): export trusted pairs as mechanism records

2026-09-05 14:40:15 +02:00

test_mutation_fixtures.py

feat(three-stage-scan): add aliased-sink mutation the regex scan misses

2026-09-04 11:50:44 +02:00

test_obligation_check.py

feat(authorization-obligations): the obligation checker, path-aware and function-local

2026-09-06 07:24:13 +02:00

test_obligation_dominance.py

feat(authorization-obligations): Dominance protocol with an order-based default

2026-09-05 19:38:04 +02:00

test_obligation_export.py

fix(learning-harness): the round-4 review findings

2026-09-06 14:03:10 +02:00

test_obligation_facts.py

feat(authorization-obligations): group sibling handlers and find consistency anomalies

2026-09-06 07:24:13 +02:00

test_obligation_intent.py

feat(authorization-obligations): hunter adjudicates intent without raising evidence

2026-09-06 07:24:13 +02:00

test_obligation_operations.py

feat(learning-harness): declared twins and a fixture-difficulty check

2026-09-06 13:09:56 +02:00

test_obligation_policy.py

feat(authorization-obligations): group sibling handlers and find consistency anomalies

2026-09-06 07:24:13 +02:00

test_obligation_reports.py

feat(authorization-obligations): reports and ranking name the obligation

2026-09-06 07:24:13 +02:00

test_obligation_scan.py

feat(authorization-obligations): run the checker in MAP with config and manifest counters

2026-09-06 07:24:13 +02:00

test_obligation_shapes.py

feat(authorization-obligations): obligation shapes derived from trusted pairs

2026-09-05 19:38:04 +02:00

test_obligation_siblings.py

feat(authorization-obligations): group sibling handlers and find consistency anomalies

2026-09-06 07:24:13 +02:00

test_opencode_skills.py

chore: add kiro spec-driven tooling and steering docs

2026-06-21 14:28:19 +02:00

test_pair_corpus.py

feat(learning-harness): family scoring on the pairs hunter path

2026-09-06 10:41:06 +02:00

test_pair_harvest.py

feat(authorization-obligations): harvest mode handler_context and the obligation label

2026-09-06 07:24:13 +02:00

test_pair_hygiene.py

fix(learning-harness): the round-4 review findings

2026-09-06 14:03:10 +02:00

test_pair_registration_context.py

feat(learning-harness): absence rows travel with their registration

2026-09-06 12:39:55 +02:00

test_pipeline_calibration.py

feat(calibration): automatic in-pipeline false-positive feedback

2026-06-21 20:44:08 +02:00

test_policy.py

feat(governance): Phase 1 — CWE policy store + project score (zero-dep)

2026-06-22 12:31:35 +02:00

test_preprocess.py

feat(scan): add initial SAST pipeline

2026-06-17 21:13:51 +02:00

test_provenance.py

feat(pair-corpus-honesty): deterministic provenance fingerprint at scan time

2026-09-05 10:45:43 +02:00

test_provider_retry.py

feat(hardening): Phase 15 — redaction, retry/backoff, budgets, threat-model docs

2026-06-22 20:02:34 +02:00

test_rank.py

feat(harness): add runtime and reachability mapping

2026-06-18 07:57:06 +02:00

test_redaction.py

feat(hardening): Phase 15 — redaction, retry/backoff, budgets, threat-model docs

2026-06-22 20:02:34 +02:00

test_regress_candidates.py

feat(three-stage-scan): select bounded regress candidates with sev-5 extras

2026-09-03 22:36:02 +02:00

test_regress_recipes.py

feat(three-stage-scan): add language recipes with inconclusive fallback

2026-09-03 22:48:10 +02:00

test_regress_safety.py

feat(three-stage-scan): reject unsafe regression snippets before execution

2026-09-03 22:12:45 +02:00

test_regress_verdicts.py

feat(three-stage-scan): map sandbox outcomes to worth-fixing verdicts

2026-09-03 22:59:01 +02:00

test_reports.py

feat(three-stage-scan): report map and honor fail-on worth-fixing

2026-09-04 00:18:33 +02:00

test_ruleset.py

feat(benchmark): Phase 2 (task 4) — actionable per-rule signals, ledger, FP knob

2026-06-22 15:41:17 +02:00

test_run.py

feat(scan): add initial SAST pipeline

2026-06-17 21:13:51 +02:00

test_sandbox_probe.py

feat(three-stage-scan): run or skip deep regression with visible degradation

2026-09-03 23:43:15 +02:00

test_sandbox_runner.py

feat(three-stage-scan): run or skip deep regression with visible degradation

2026-09-03 23:43:15 +02:00

test_scan_deep.py

feat(propose-adjudicate-prove): overlay adjudicates inventory before prove

2026-09-04 13:57:33 +02:00

test_scan_pipeline_integrity.py

feat(three-stage-scan): attach tool hunter to map hotspots

2026-09-04 00:04:36 +02:00

test_score_pipeline.py

feat(ruleset): Phase 2 (task 3) — rules as governed data, severity from policy

2026-06-22 15:27:16 +02:00

test_scoring.py

feat(benchmark): Phase 2 (task 4) — actionable per-rule signals, ledger, FP knob

2026-06-22 15:41:17 +02:00

test_semantic_cst.py

feat(tree-sitter-overlay-extra): parse JS/C/Java overlay via optional extra

2026-09-04 15:07:35 +02:00

test_semantic_extra.py

feat(tree-sitter-overlay-extra): parse JS/C/Java overlay via optional extra

2026-09-04 15:07:35 +02:00

test_semantic_overlay.py

feat(tree-sitter-overlay-extra): parse JS/C/Java overlay via optional extra

2026-09-04 15:07:35 +02:00

test_semantic_pipeline.py

feat(propose-adjudicate-prove): overlay adjudicates inventory before prove

2026-09-04 13:57:33 +02:00

test_skills.py

feat(skills): Phase 14 — Trail of Bits skill index + router

2026-06-22 19:51:04 +02:00

test_slot_contract.py

feat(slots): Phase 3 task 6.3 — model-disabled, slot-contracted deterministic stages

2026-06-22 18:39:09 +02:00

test_split_sink_fixtures.py

feat(three-stage-scan): add split-sink fixtures the regex scan misses

2026-09-04 11:49:44 +02:00

test_split_sink_regress.py

feat(three-stage-scan): prove split-sink triggerable in sandbox

2026-09-04 11:54:01 +02:00

test_stages.py

feat(three-stage-scan): keep quick as stage-1 inventory

2026-09-03 23:14:00 +02:00

test_tool_hunter.py

fix(learning-harness): a hunt that ends in prose is asked to format it

2026-09-06 14:18:09 +02:00

test_typescript_grammar.py

feat(learning-harness): absence rows travel with their registration

2026-09-06 12:39:55 +02:00

test_variant_scan.py

feat(corpus-seeded-mechanisms): reports name the mechanism, its known fix guard and pair provenance

2026-09-05 15:35:08 +02:00

test_variant_search.py

refactor(corpus-seeded-mechanisms): apply review round 2 suggestions

2026-09-05 14:55:00 +02:00

test_variants.py

feat(corpus-seeded-mechanisms): structural shapes derived from labeled pairs

2026-09-05 14:27:33 +02:00

test_verification.py

ci: add quality gates and release workflows

2026-06-18 10:06:21 +02:00

test_verify_judge.py

feat(harnessx): configurable provider + per-task budget caps

2026-06-22 17:59:44 +02:00