CI runs ruff format --check and ruff check over the whole repository; 29 files had drifted from the formatter and
four measurement scripts carried lint findings, so every push since 25ae4b4 failed before pytest ran.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Three builds: the first two read 150 rows as empty (a producer bug, fcb43fd), the third is clean. The diff
producer was blind to removed lines (537b61e). Screen: deletion_count 0.89 is the reversed-fix construction;
on rows whose file lost lines entry_point_count 0.83 and caller_count 0.73 are the class signal; semantic covers
262 rows (TypeScript only). Repository folds assigned, leak audit recorded, manifest ac-ablation-1 registered
for decision (five arms, about 35 dollars).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
DSPy baseline: 2/5 pushes, 4/48 locations, 14 of 90 ordinary pushes flagged, 25 suspicions (3 true), 11 invalid-request
failures in shopper and trigger.dev, $6.13. Pilot hunter: 2/5 pushes, 4/48 locations, 99 of 100 ordinary pushes
flagged, 594 findings (3 true). Different pushes caught; nothing in trigger.dev.
The runner booked usd 0.0 per change so --budget-usd never stopped a run; it now prices every LM call of the change
at the registered prices and writes usd into the record.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The 150 unread rows were not a checkout failure: the entry-point scan read every product file and one empty
file raised empty_source, which escaped the cached property and failed the source producer on every row of
the change (and re-read the whole tree per row). Empty product files are skipped; the row's own file still
has to be non-empty.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The training clones are sparse-checkout and a worktree inherits it; without this the hunter arm raises
unreadable_hotspot on every change of the access-control pool. Same helper and checkout proof as the hunt runner.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Two instrument findings from the first access-control feature table (952 rows, 74 positive rows):
1. A fix that adds a guard is, replayed base-to-head, a push that removes it. The diff producer only kept
hunks with added lines at head, so 38 of 74 positive rows had no hunk in their file at all and 69 of 74
were "not on a changed line", while 60% of ordinary rows were. The diff annotation separated the labels
by construction and was blind to exactly the access-control shape. The producer (version 2) now records
per file the head lines after which the push removed code; distance and on_changed_line count them, the
model's hunks input renders them as removed_after_line, and diff.deletion_count is an engineered feature.
2. 150 rows (16%, every positive of one repository) read empty files from the worktree although git holds
the blobs; the build ran while another process checked out from the same clone. _full_checkout now
proves the checkout: git status must be clean, a dirty tree is restored once, then InstrumentFailure
checkout_incomplete. The table script cross-checks every unread row against the blob size and refuses
to screen a partly unread tree, and records the diff-scoped recall ceiling (positive rows a label-blind
enumeration would reach).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Java 63 of 69 sites. verify: 148 records, 701 deepseek-flash calls, no provider errors. The 21
uncloned sources are in sources_access_control_dropped.json for a later clone pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
sources --family <family> keeps only advisories whose CWE the taxonomy maps to that family, writes
sources_<family>.json beside the training set, and excludes the pilot, every repository the
benchmarks already name, the strict used set and every earlier batch; --prefilter-only runs the
dump pass without GitHub. Implemented by Codex (gpt-6-astra), gated here: names normalize rejects
exclude nothing instead of aborting, and the strict used set is in the exclusions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
label_rows honoured only a change's primary site, so 43 of the 48 accepted access-control
locations were unlabelled; every entry of targets is now a positive with its own label_ref and
span. The training clones were emptied with sparse-checkout to save disk and a new worktree
inherits it: every file read failed and the source and semantic producers reported an unread tree
on all 599 rows. The runner now disables sparse-checkout in the worktree it adds.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
mcp-documentation-server (8 ordinary commits) and line-desktop-mcp (13) are left out of this freeze;
their locations stay accepted for a later one.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ordinary_history(limit=) walks only the newest commits; assemble.freeze(history_limit=) and
--history-limit pass it through and the record carries it. The held-out benchmark passes none.
Reason: trigger.dev's 7,109-commit walk runs for hours at the per-commit cost of the function
and generated checks; the training freeze now walks its newest 1,500.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
rounds.py runs a candidate over compile or held-out folds through runner.run, so a tool-using
program has its tools bound to each change's checkout, and turns paired records into the
FoldResults accept.decide reads (closes the gap noted in bacd7a1). report.py runs an accepted
program once over the held-out pilot: a ledger line reserved before the run refuses a second run
per program id, contaminated or unchecked training runs are refused, the instrument is proved per
change and more than 5% failures makes the run invalid with no numbers quoted, the pilot's own
scorer and metrics give the after numbers, the recorded hunter numbers give the before, baseline
repeats pair with the single run, the proof arm's workload per 100 pushes is counted, and the gate
distance is read off the confidence curve. 124 hunt and stats tests pass. Implemented by Codex
(gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
propose.py: one change per round (instruction, demonstrations or one mask), examples built from
compile folds only with a provenance record of the change ids used, a registry for GEPA, MIPROv2
and BootstrapFewShot (MIPROv2 without the data-aware proposer so labels are never serialised),
compile-fold scoring with Wilson and the repository-paired bootstrap, and a compile-only history
the next round may read. accept.py: the rule as independent clauses (both folds non-negative with
one strictly positive, per-family regression budget on held-out recall, tie tolerance, no
acceptance without an interval, instruction cap, one change only), the comparison recorded in the
candidate artifact's configurations, rejections logged without held-out numbers. 48 fixture tests.
Known gap for live rounds: the optimisers and propose._score call the program directly, so a
tool-using program must be scored through the runner with its tools bound per checkout (6.1).
Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
One append-only JSONL per clone (named by the sha256 of its top-level path), one line per commit
with the row as the walk built it and HISTORY_CACHE_VERSION, flushed after every commit so a
killed run keeps its progress. A cached row is reused only when its parents match the current log;
other versions and torn lines are recomputed. With no cache directory nothing changes. The
7,400-commit trigger.dev walk that took over an hour per freeze becomes a one-time cost.
Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
contamination.py reads the repositories named by the held-out pilot manifest and by every
benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks
each run directory clean or contaminated with the report digest and time, and refuses to count
an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is
contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so
the package imports without the benchmarks tree.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ablation.py registers the manifest (table digest, arms as masks, folds, metric, sample, budget,
rule, seed, program id) before any run and refuses a run or an analysis that does not match it.
Arms run over the same changes through runner.run, paired by change; effects on recall and on
suspicions per ordinary push are reported pooled and per family with the repository-clustered
bootstrap and exact McNemar, plus the recall-against-suspicions curve at every reported
confidence (never a tuned cut). The importance table and decisions read held-out effects only;
a neutral annotation survives only when a kept one needs it. 30 fixture tests with a scripted
model. Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The held-out benchmark keeps the 40-commit floor and the union of twelve-commit stratified draws.
A training freeze passes floor=20 (never below the ratio) and the record carries ordinary_floor.
When the union of twelve-commit draws cannot reach the count (small cells stay unallocated on
every draw), one stratified draw of the full count is used; with count equal to the eligible
commits that is every one of them.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sourced apart from the 12 pilot repositories: 212 candidates (pool-fresh, pool-retry, OSV for PyPI,
npm, Maven and Packagist), 49 dropped because the benchmarks already name their repository, 413
catalog locations, 99 passing all three verifier checks in 28 repositories. Short of the 40-positive
target. catalog.py now reads .java and .php. Freeze waits on the maintainer's spot-check
(CANDIDATES.md, SOURCING.md).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The full suite after the replay found regressions from 062a7b7 and 94e96c6:
- catalog_gen and candidates.labeled_spans still unpacked 3-tuples and expected None on a parser
rejection; source.ranges returns FunctionRange (4 fields, module entry first) and raises ValueError.
- ranges_from_ir passed the language to declarator_name, so the repo-facts declaration table named a
callback call (exec(..., function () {) as a function; the pre-P1 call had no language.
- the catalog_gen fail-loud test blocked the old module name.
- docs: a renumbered examples anchor, and two .kiro paths in the public threat model.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
DSPy 3.4.1 raises AdapterParseError for an unparseable ReAct step, which is not the ValueError its loop
catches, so one malformed step discarded the whole hunt (7 of 60 hunts in the repeats). The step now ends
the loop like an invalid tool choice. Records carry loop_parse_failed beside parse_failed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Committed before the repeats finish. Unit is (change, repeat); paired bootstrap
over changes and exact McNemar from openultrasast.stats; failures stay in the
denominator. Dry run on the existing single runs reproduces 6/12 and 3/12.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
3 of 12 with the fixed answer check against the pilot hunter's 6 of 12, one
run each, not significant. The intended zero-cost replay was not a replay:
answers at temperature 0 differ between runs and the cache did not serve the
second run. Programs must be compared over repeated runs, paired by change.
About $1.10 for both runs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The answer check counted only the rendered function text as shown, so 7 of
8 baseline suspicions citing lines the model had read with read_file were
marked unknown (baseline 1/12 against the pilot hunter's 6/12). Tools now
record the lines they return. A DSPy parse failure scores as zero findings
with a flag instead of a broken run. The per-run response cache now takes
effect through dspy.configure_cache.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 of 12 positives; 12.6 false alarms per 100 ordinary pushes (2.1 on changed
lines only). The misses come from what the rules and facts express (access
control lives in obligations, which do not convert; sources cover request
objects only), not from the engine. Comparison arm only: a Semgrep miss is
never evidence against a hunter finding.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
A Semgrep, quick-rule or engine match may enter the feature table as
positive evidence; its absence is never a feature, filter, demotion or
gate on a hunter suspicion, and no acceptance rule counts it. The Semgrep
arm is a comparison only and may at most replace the regex quick tier.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
openultrasast.stats: Wilson intervals (statsmodels), exact McNemar, a
repository-clustered percentile bootstrap and its paired form with one
quantile convention, grouped folds (GroupKFold, leave-one-group-out), and
per-column separation (ROC AUC) for the leak audit and the signal screen.
Checked against the benchmark scorer's Wilson and the learn harness's
McNemar. hunt/features.py turns the feature table into the engineered
frame (distances, counts, flags) and keeps the map from every engineered
feature to the annotation the model is shown, so an ablation mask drops
both, and screen() says which annotation carries signal before a model
call is spent. The [hunt] extra now carries the libraries; the core
install is unchanged. The learn harness's own copies stay until that line
is revived.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
hunter-learning-loop groups 1-3 (first increment). The feature table is the
contract: one row per touched function with diff, source, semantic and
taxonomy columns, provenance per producer, labels by the benchmark's rule.
Producers read the repository at head only (a planted advisory id and
commit text never reach a feature). The hunt program is a DSPy ReAct loop
over a signature whose inputs are the unmasked columns in a fixed order;
answers outside the closed family list or citing unseen lines score as
unknown. The runner writes records the benchmark scorer reads, resumable
by change, program id and mask, and refuses the held-out pilot unless the
run is a report run. Research extra [hunt] = dspy, pandas; the core
install is unchanged. 43 tests on a scripted model, no network.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Engine LGPL-2.1, shelling out carries no obligation; the registry rules are
under the Semgrep Rules License since 2024-12-13 and cannot ship, only our
own converted rules can. Measured 4.7 s startup and 5 to 10 ms per file;
converted regex and taint rules fire for Python, PHP and JavaScript; C
propagation and Groovy do not; one sanitizer semantic we refuse on purpose
is accepted by Semgrep. Recommendation: a Semgrep arm on the valid pilot
first. Recorded in the hunter-learning-loop research note and breadth.toml.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
610 pair files, 570 labelled functions. lizard matches the parser engine on
C, Python and Java but loses JavaScript names and TypeScript end lines:
keep ours, take its cyclomatic complexity as a feature candidate.
tree-sitter-language-pack gives identical spans, is faster, replaces six
grammar wheels with one and adds Go, Ruby, PHP and Perl: adopt as grammar
packaging. Also found: .c files should try the c grammar before cpp.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Six groups: table and producers; the hunter as a program with the search
metric and the run artifact; runner integration and a baseline on the 12
pilot positives; training set, folds, paired ablation and the importance
table; proposals and the acceptance rule; the pilot gain report.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Feature engineering is the centre: a feature table with switchable columns
is the contract between producers (source.facts_for, semantic facts, entry
points, diff hunks; lizard and Semgrep only after their evaluations), the
ablation runner and the hunter program. DSPy runs the program (ReAct over a
HuntSignature; GEPA and MIPROv2 propose), our rule accepts on held-out
folds. The pilot stays held out; the hook, its client and the core install
are untouched. Requirements approved by the maintainer ('4 first').
Co-Authored-By: Claude Fable 5.1 <[email protected]>
src/openultrasast/source.py replaces the copies in semantic/functions.py,
plane/tasks/repo_facts.py, learn/features.py, learn/excerpt.py and
plane/tasks/verify.py. Two engines share one sentinel (<module>), one
DECLARATION/KEYWORDS/RESERVED set and one alias map: the parser engine
(ranges, enclosing, bounds, call_sites, callers_of, facts_for) for the
scorer, the overlay and the hunter's annotations; the line-pattern engine
(declared_functions, enclosing_at, function_span, function_bounds, the brace
matcher) that labels, features, roles, excerpts and plane facts were built
on. learn no longer imports plane.tasks. Frozen records that store the old
sentinel are mapped on read (module_function).
Gate: labels rebuilt before and after are identical, 7,921 rows, after
sentinel normalisation (benchmarks/measurements/2026-10-13-source-module-
label-diff/record.json). 490 targeted tests pass; 3,312 collect. Net: 657
lines deleted, 212 added. benchmarks/unseen keeps <global> for its own
frozen manifests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: the four hunter tools no caller enables (read_definition,
entry_points, flows, obligations) with their schemas and dispatch, the dead
mapping branch of find_refs; 20 functions whose only caller was one test
(mapping task records, compute_ranking_metrics, build_quick_hunter_prompt,
validate_evidence_transition, should_fuse, harness processors,
rank_targets_with_model, model/report slice rendering, taxonomy
hierarchical/related, config_value.verdict, taint.verdict, roles.load_roles,
features.language_of and build_for_scan, folds.leave_one_*,
features.recorded_case, search BatchExecutor and the three __main__ shims,
delivery.put_input); the receiver delivery mode of the plane. push_inputs.py
moved to benchmarks/push. memory.purge stays: the live-S3 test cleanup uses
it. Net: 2,730 lines deleted, 207 added. 473 targeted tests passed; 3,282
tests collect. breadth.toml records the removals.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep
now records a regress_retired degradation and equals standard; proof is the
search-with-proof arm's job), the sandbox docker runner and probe that only
regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers
only regress called; skills.py and index.py with the hunter pool's skill
routing; model/audit.py; cpg/queries/vocabulary.sc with
benchmarks/push/ontology_gap.py. Docs and the scan skill describe the
retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed,
28 skipped; 3,331 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/improve/ (1,175 lines), the ousast improve
subcommand, plane/tasks/loop.py and loop.yaml with the --loop generation,
the memory-snapshot step in memory.seed, pairs.build_pair_signals, and the
four improve tests plus the loop cases in other tests. Docs no longer
describe the lever. Net: 3,073 lines deleted, 83 added. Targeted tests:
154 passed; 3,436 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
PATHFINDER-2026-10-13/: feature inventory, ten per-unit flowcharts with
file:line evidence, cross-feature and within-unit duplication reports, a
proposal that keeps every package on one of the four active lines, and
copy-ready plan prompts in order. .kiro/breadth.toml: every spec and
capability with a status and the deletion that closes it. No code changed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
fossil-mcp dead-code over src found 72 candidates; 69 are referenced by tests,
__all__ or other modules, 3 are not and are removed (cli._first_source,
pairs._scorable, provider.openrouter._extract_message_content). Eight spec
directories that steering already called retired or superseded leave the tree
(history keeps them, commits named in overview.md): harnessx-removal,
harnessx-self-improving-rulesets, openrouter-sast-harness, learning-harness,
constrained-detector, three-stage-scan, real-world-vfc-slice,
unseen-repo-evaluation. overview.md gets a current-state section: the valid
benchmark numbers, what they say, and the four active lines in order.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The hunter gets an annotation model (named, versioned facts about the push)
and statistical ablation decides which annotations stay. Labels from fix
sites and ordinary pushes, training set apart from the held-out pilot,
compiled replayable program, train-and-holdout acceptance rule, two stages
reported separately, no thresholds set in this spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sweep over the pilot's hunter records: most 'vulnerable' suspicions on ordinary
pushes sit far from the diff. Scope to the change, then confidence, then the
config_secrets family; 5/12 recall kept at 8.8 false alarms per 100 (in-sample).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opt-in filter on the classifier verdict for hunter-sourced defects; the default
keeps everything. Each combined record notes the kept verdicts and the counts.
Live check on the pilot: (a)+(b) filtered to 'vulnerable' gives recall 5/12 and
54.4 false alarms per 100 ordinary pushes, matching the scratch numbers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Arm (b) hunter over all 252 changes: recall 6/12 raw, 5/12 with the classifier
filter; false alarms 92.5 and 54.4 per 100 ordinary pushes. Arm (a) push hook at
163/252: recall 0/12, 0.7 false alarms per 100. Per-positive table, classifier
stats, the one hook false alarm, and the disk fix for the replay output.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Live run: every classification failed with ValueError because Caller.ask
reads client.usage as a dict (the learn CLI always hands it a MeteredClient)
while a bare DeepSeekChatClient keeps usage as a list of rows. build_caller
now wraps a client that carries a usage list in MeteredClient; scripted and
already-metered clients pass through unchanged.
Regression test runs classify_family through the real DeepSeekChatClient
wrapper with a stub transport (the module fixture that stubs classify is
restored for it). The error field stays the exception type only, since a
message can carry a model key.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.
hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.
14 new offline tests, 67 green with the benchmark tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
