CI runs ruff format --check and ruff check over the whole repository; 29 files had drifted from the formatter and
four measurement scripts carried lint findings, so every push since 25ae4b4 failed before pytest ran.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Three builds: the first two read 150 rows as empty (a producer bug, fcb43fd), the third is clean. The diff
producer was blind to removed lines (537b61e). Screen: deletion_count 0.89 is the reversed-fix construction;
on rows whose file lost lines entry_point_count 0.83 and caller_count 0.73 are the class signal; semantic covers
262 rows (TypeScript only). Repository folds assigned, leak audit recorded, manifest ac-ablation-1 registered
for decision (five arms, about 35 dollars).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
DSPy baseline: 2/5 pushes, 4/48 locations, 14 of 90 ordinary pushes flagged, 25 suspicions (3 true), 11 invalid-request
failures in shopper and trigger.dev, $6.13. Pilot hunter: 2/5 pushes, 4/48 locations, 99 of 100 ordinary pushes
flagged, 594 findings (3 true). Different pushes caught; nothing in trigger.dev.
The runner booked usd 0.0 per change so --budget-usd never stopped a run; it now prices every LM call of the change
at the registered prices and writes usd into the record.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The training clones are sparse-checkout and a worktree inherits it; without this the hunter arm raises
unreadable_hotspot on every change of the access-control pool. Same helper and checkout proof as the hunt runner.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Two instrument findings from the first access-control feature table (952 rows, 74 positive rows):
1. A fix that adds a guard is, replayed base-to-head, a push that removes it. The diff producer only kept
hunks with added lines at head, so 38 of 74 positive rows had no hunk in their file at all and 69 of 74
were "not on a changed line", while 60% of ordinary rows were. The diff annotation separated the labels
by construction and was blind to exactly the access-control shape. The producer (version 2) now records
per file the head lines after which the push removed code; distance and on_changed_line count them, the
model's hunks input renders them as removed_after_line, and diff.deletion_count is an engineered feature.
2. 150 rows (16%, every positive of one repository) read empty files from the worktree although git holds
the blobs; the build ran while another process checked out from the same clone. _full_checkout now
proves the checkout: git status must be clean, a dirty tree is restored once, then InstrumentFailure
checkout_incomplete. The table script cross-checks every unread row against the blob size and refuses
to screen a partly unread tree, and records the diff-scoped recall ceiling (positive rows a label-blind
enumeration would reach).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Java 63 of 69 sites. verify: 148 records, 701 deepseek-flash calls, no provider errors. The 21
uncloned sources are in sources_access_control_dropped.json for a later clone pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
sources --family <family> keeps only advisories whose CWE the taxonomy maps to that family, writes
sources_<family>.json beside the training set, and excludes the pilot, every repository the
benchmarks already name, the strict used set and every earlier batch; --prefilter-only runs the
dump pass without GitHub. Implemented by Codex (gpt-6-astra), gated here: names normalize rejects
exclude nothing instead of aborting, and the strict used set is in the exclusions.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
mcp-documentation-server (8 ordinary commits) and line-desktop-mcp (13) are left out of this freeze;
their locations stay accepted for a later one.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ordinary_history(limit=) walks only the newest commits; assemble.freeze(history_limit=) and
--history-limit pass it through and the record carries it. The held-out benchmark passes none.
Reason: trigger.dev's 7,109-commit walk runs for hours at the per-commit cost of the function
and generated checks; the training freeze now walks its newest 1,500.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
One append-only JSONL per clone (named by the sha256 of its top-level path), one line per commit
with the row as the walk built it and HISTORY_CACHE_VERSION, flushed after every commit so a
killed run keeps its progress. A cached row is reused only when its parents match the current log;
other versions and torn lines are recomputed. With no cache directory nothing changes. The
7,400-commit trigger.dev walk that took over an hour per freeze becomes a one-time cost.
Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The held-out benchmark keeps the 40-commit floor and the union of twelve-commit stratified draws.
A training freeze passes floor=20 (never below the ratio) and the record carries ordinary_floor.
When the union of twelve-commit draws cannot reach the count (small cells stay unallocated on
every draw), one stratified draw of the full count is used; with count equal to the eligible
commits that is every one of them.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sourced apart from the 12 pilot repositories: 212 candidates (pool-fresh, pool-retry, OSV for PyPI,
npm, Maven and Packagist), 49 dropped because the benchmarks already name their repository, 413
catalog locations, 99 passing all three verifier checks in 28 repositories. Short of the 40-positive
target. catalog.py now reads .java and .php. Freeze waits on the maintainer's spot-check
(CANDIDATES.md, SOURCING.md).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The full suite after the replay found regressions from 062a7b7 and 94e96c6:
- catalog_gen and candidates.labeled_spans still unpacked 3-tuples and expected None on a parser
rejection; source.ranges returns FunctionRange (4 fields, module entry first) and raises ValueError.
- ranges_from_ir passed the language to declarator_name, so the repo-facts declaration table named a
callback call (exec(..., function () {) as a function; the pre-P1 call had no language.
- the catalog_gen fail-loud test blocked the old module name.
- docs: a renumbered examples anchor, and two .kiro paths in the public threat model.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Committed before the repeats finish. Unit is (change, repeat); paired bootstrap
over changes and exact McNemar from openultrasast.stats; failures stay in the
denominator. Dry run on the existing single runs reproduces 6/12 and 3/12.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
3 of 12 with the fixed answer check against the pilot hunter's 6 of 12, one
run each, not significant. The intended zero-cost replay was not a replay:
answers at temperature 0 differ between runs and the cache did not serve the
second run. Programs must be compared over repeated runs, paired by change.
About $1.10 for both runs.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
0 of 12 positives; 12.6 false alarms per 100 ordinary pushes (2.1 on changed
lines only). The misses come from what the rules and facts express (access
control lives in obligations, which do not convert; sources cover request
objects only), not from the engine. Comparison arm only: a Semgrep miss is
never evidence against a hunter finding.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Engine LGPL-2.1, shelling out carries no obligation; the registry rules are
under the Semgrep Rules License since 2024-12-13 and cannot ship, only our
own converted rules can. Measured 4.7 s startup and 5 to 10 ms per file;
converted regex and taint rules fire for Python, PHP and JavaScript; C
propagation and Groovy do not; one sanitizer semantic we refuse on purpose
is accepted by Semgrep. Recommendation: a Semgrep arm on the valid pilot
first. Recorded in the hunter-learning-loop research note and breadth.toml.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
610 pair files, 570 labelled functions. lizard matches the parser engine on
C, Python and Java but loses JavaScript names and TypeScript end lines:
keep ours, take its cyclomatic complexity as a feature candidate.
tree-sitter-language-pack gives identical spans, is faster, replaces six
grammar wheels with one and adds Go, Ruby, PHP and Perl: adopt as grammar
packaging. Also found: .c files should try the c grammar before cpp.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
src/openultrasast/source.py replaces the copies in semantic/functions.py,
plane/tasks/repo_facts.py, learn/features.py, learn/excerpt.py and
plane/tasks/verify.py. Two engines share one sentinel (<module>), one
DECLARATION/KEYWORDS/RESERVED set and one alias map: the parser engine
(ranges, enclosing, bounds, call_sites, callers_of, facts_for) for the
scorer, the overlay and the hunter's annotations; the line-pattern engine
(declared_functions, enclosing_at, function_span, function_bounds, the brace
matcher) that labels, features, roles, excerpts and plane facts were built
on. learn no longer imports plane.tasks. Frozen records that store the old
sentinel are mapped on read (module_function).
Gate: labels rebuilt before and after are identical, 7,921 rows, after
sentinel normalisation (benchmarks/measurements/2026-10-13-source-module-
label-diff/record.json). 490 targeted tests pass; 3,312 collect. Net: 657
lines deleted, 212 added. benchmarks/unseen keeps <global> for its own
frozen manifests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: the four hunter tools no caller enables (read_definition,
entry_points, flows, obligations) with their schemas and dispatch, the dead
mapping branch of find_refs; 20 functions whose only caller was one test
(mapping task records, compute_ranking_metrics, build_quick_hunter_prompt,
validate_evidence_transition, should_fuse, harness processors,
rank_targets_with_model, model/report slice rendering, taxonomy
hierarchical/related, config_value.verdict, taint.verdict, roles.load_roles,
features.language_of and build_for_scan, folds.leave_one_*,
features.recorded_case, search BatchExecutor and the three __main__ shims,
delivery.put_input); the receiver delivery mode of the plane. push_inputs.py
moved to benchmarks/push. memory.purge stays: the live-S3 test cleanup uses
it. Net: 2,730 lines deleted, 207 added. 473 targeted tests passed; 3,282
tests collect. breadth.toml records the removals.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep
now records a regress_retired degradation and equals standard; proof is the
search-with-proof arm's job), the sandbox docker runner and probe that only
regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers
only regress called; skills.py and index.py with the hunter pool's skill
routing; model/audit.py; cpg/queries/vocabulary.sc with
benchmarks/push/ontology_gap.py. Docs and the scan skill describe the
retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed,
28 skipped; 3,331 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Sweep over the pilot's hunter records: most 'vulnerable' suspicions on ordinary
pushes sit far from the diff. Scope to the change, then confidence, then the
config_secrets family; 5/12 recall kept at 8.8 false alarms per 100 (in-sample).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Opt-in filter on the classifier verdict for hunter-sourced defects; the default
keeps everything. Each combined record notes the kept verdicts and the counts.
Live check on the pilot: (a)+(b) filtered to 'vulnerable' gives recall 5/12 and
54.4 false alarms per 100 ordinary pushes, matching the scratch numbers.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Arm (b) hunter over all 252 changes: recall 6/12 raw, 5/12 with the classifier
filter; false alarms 92.5 and 54.4 per 100 ordinary pushes. Arm (a) push hook at
163/252: recall 0/12, 0.7 false alarms per 100. Per-positive table, classifier
stats, the one hook false alarm, and the disk fix for the replay output.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Live run: every classification failed with ValueError because Caller.ask
reads client.usage as a dict (the learn CLI always hands it a MeteredClient)
while a bare DeepSeekChatClient keeps usage as a list of rows. build_caller
now wraps a client that carries a usage list in MeteredClient; scripted and
already-metered clients pass through unchanged.
Regression test runs classify_family through the real DeepSeekChatClient
wrapper with a stub transport (the module fixture that stubs classify is
restored for it). The error field stays the exception type only, since a
message can carry a model key.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.
hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.
14 new offline tests, 67 green with the benchmark tests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.
`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.
9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
First live run of the valid detection benchmark. Labels come from the fix
site (fix^1), never SZZ.
- 32 held-out Python/JS advisories sourced from the public pool-p1 set,
CWE + text from OSV. 133 candidate locations -> 125 after dropping
tests/minified.
- 125 DeepSeek verification calls: 31 locations passed all three checks
(vulnerable pre-fix, fix addresses it, reachable in-repo) across 12 repos.
- Maintainer spot-check accepted all 12.
- 20 ordinary negatives per positive change from the same repos, 240 total.
- Held-out guard: 0 dropped against the training-only used set.
- Frozen manifest at benchmarks/valid/pilot/ (252 changes), provenance and
NOTES under benchmarks/measurements/2026-10-12-valid-benchmark-pilot/.
Three fixes found by the live run:
- catalog.py: skip a site the fix renamed or removed instead of aborting.
- assemble.py: negative ratio counts positive changes, not accepted
functions (malla would have demanded 260 negatives). Test updated.
- held-out guard: run against training sources only; build_used() also
scans measurements/populations, which flagged the 12 repos via my own
SZZ-pool analysis outputs (circular).
Replay and metrics not yet run; see NOTES.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.
Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.
New offline modules under benchmarks/valid/:
- catalog.py positives labelled at the fix site (never introducing/last_touch)
- verify.py one LLM call per case (vulnerable? fix addresses it? reachable
in-repo?) + human spot-check; this is the app-shaped filter.
Honors the instrument rule: bytes read per function, reject on
an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
the held-out guard
- metrics.py precision (PPV), false alarms per 100 pushes, exploit-verified
rate, and the plain headline, layered on score.py
36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roadmap "Where we are" table said Baseline (G0.2) = "Not yet, run on kube" -- stale.
Record 07 (G0 exit) was completed right after on the docker lane (e901640, 1300/1300,
catch 3.3% / FP 3.1%, regex/quick-rule floor, engine_covered=0). G0 is exited; the kube
`ax` lane was never needed for the baseline. Reframed the readiness plan: steps 1-4 done,
the `ax` lane moves to G1 (engine arm at scale, gated by spec tasks 4.4/5.4).
Also fixed the ax binary in the coordinator commands: use $HOME/go/bin/ax, NOT
$HOME/.local/ax-v0.3.1/ax (the latter hits the AX-lane dispatch blocker recorded in the
unseen spec tasks.md). README.md + benchmarks/unseen/README.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
test_no_pair_repeats_a_repository_the_benchmarks_already_name failed: advisory-fix
catalog repos also appeared elsewhere under benchmarks/. Root cause split two ways:
1. Genuine leak: 16 catalog repos are also subjects in benchmarks/search/private/
pilot.json (a real private pilot population). Removed those 16 pairs from the
three catalogs and their verification.json rows (totals recomputed, catalog ==
verification preserved).
2. Spurious: the overlap scan greped ~49k vendored files under
benchmarks/search/private/runs/*/checkout/node_modules/, so any catalog repo a
vendored dependency's README names collided. Added a narrow vendored-tree skip
(node_modules/vendor) to the scan rather than deleting labelled pairs over a
dependency mention.
Implemented from Codex's first pass (gpt-6.1-sol), then gated and corrected: Codex
removed 20 pairs; 4 of those (backstage/backstage, koajs/koa, beaugunderson/
ip-address, cakephp/cakephp) were driven only by vendored/non-population mentions
and are kept. Also excluded advisory-fixes-spread.json (catalog-derived report)
and research-2026-09-26.md (selection write-up) as non-population.
tests/test_advisory_fix_pairs.py + tests/test_learn_labels.py: 53 passed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
taint.request_params emits guards/fixedOrigin/originAnchors/quotedSanitizers/
fieldParameterSources, which taint.sc reads from per-request JSON in its BATCH
branch but does NOT declare on `@main def exec` and does NOT consume in its
single-request branch (it defaults them). The single-query path passed them as
`--param guards=...` (disposable) or named args (session), so joern errored
`Unknown arguments` and every cpg.run("taint", ...) failed -- a silent zero of
the shape AGENTS.md warns about. scan_region->taint_verdict->_usable_flows and
the pair-path verdict()/verdicts() were dead; the push/scan batch route was
unaffected (it carries the extras as JSON the script parses itself).
Fix: parse the script's `@main def exec(...)` signature once (_declared_exec_params
+ memoised _declared_params) and, in query(), filter params to the declared set
before building either route. Faithful, not lossy -- the single-request branch
could never use those 5 fields. No-op for config/dominance (their request_params
emit only declared params; verified). Only taint sheds the 5 extras.
Verified end-to-end on banks (patched backend in the replay image): single-query
path now runs; baseline 0 findings, annotated search 2 path/model_entailed
findings (prompt->write_text). Regression test asserts no undeclared param
reaches the command line and the query parses, with a fake runner (no Joern).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.
Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran ousast scan --mode deep on the 5 app-shaped + sound pool targets (the fair change-scoped
eval set from the shape classification). Detection 3/5: graphiti (hunter, Cypher injection),
mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE overlay -- 'outbound.request reached
from parameter' on readFileFromUrl; parameter-as-source fired because an entry point is present).
2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families families.toml
documents as limited.
This validates the strategic pivot: the near-zero catch was substantially a TARGET MISMATCH --
on app-shaped repos the engine itself detects (0 -> 3/5). But all hits stop at suspicion/
static_corroboration; release-grade BLOCK still needs proof-arm elevation + family coverage +
the DSPy hint loop. Cost ~$0.007/repo. So bet #1 (fix eval target) is necessary and material
but not sufficient.
Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat,
DjanGoat, vibe-py); measure proof-arm suspicion->proven lift.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Named the core problem with the maintainer: the tool spots dangerous ops but cannot prove
attacker-reachability -- the trust boundary lives outside the changed library code. Chosen
strategic bet (of three): FIX THE EVALUATION TARGET FIRST.
LLM shape classification of the 49 public pool vulns ($0.033): reachability_in_repo only
24% (12/49); 76% are pure libraries. A legitimate change-scoped catch needs sound label AND
reachability in-repo -> just 5/49 (~10%): graphiti, desktopcommandermcp, malla,
document-merge-service, mcp-remote. Shape predicted the hunter outcome (graphiti app-shaped ->
caught; bedrock library -> missed). So record-07's 3/92 is scored against a ~90%-illegitimate
denominator -- the near-zero catch is largely a target mismatch.
Next: deep scan on the 5 fair targets for a per-tier catch rate; then an app-shaped eval set
from known-vulnerable apps. Artifacts: 2026-10-12-pool-p1-shape-classification/ (+ earlier
label-audit, hunter-sound-targets). Memory: evaluation-target-mismatch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed:
- graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces
build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the
labelled function, location-matches). The static engine produced zero findings here.
- bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never
named the install_packages -> pip command injection.
Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything
stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs
proven findings.
Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO,
pluggable; evolving field) whose objective is to learn what hints/annotations steer the model
to the real sink, sourced from the framework knowledge already shipped (no new user
annotation), with LLM-proposed boundaries graduating into the facts.
Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen
the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek
(advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%)
sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped
hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7,
output_encoding 1/8, config_secrets 1/4.
Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta
hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES).
Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine
missed both = the real detection gaps for the LLM-guided hunter.
Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek
is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The real bottleneck was not enumeration or extraction: canonicalize_used
resolves the entire exclusion set (measured: 4,499 names) via serial
GitHub get_repo on every launch, before any harvesting -- ~30 min of
network-latency-bound work that also burns ~4,499 of GitHub's 5,000/hr
budget before real work starts (hence the low CPU and the "never
finishes"). The earlier resolution-cache only banked via
BankedGitHub.resolve, but canonicalize_used calls eligibility.resolve ->
get_repo directly, so it was never cached and re-ran every time.
Cache get_repo at the real choke point. CorpusBank gains cache_repo /
get_cached_repo storing one object per normalized name at
<prefix>/repos/<sha256(name)>.json -- NOT seed-scoped, since repo
metadata is seed-independent, so every seed/run shares it. 404s are
cached as None too (a found-but-None hit) so known-missing repos are not
re-fetched. BankedGitHub.get_repo now checks the local journal, then the
bucket, and only then GitHub (banking the result). canonicalize_used,
advisory_index and candidate resolution all funnel through get_repo, so
all of them are covered.
Effect: the first run pays the lookups once and banks them; every later
launch reads them from the bucket (no GitHub, no rate-limit spend), so
the rate budget goes to actual candidate harvesting and restarts are
cheap. Proven live against ousast-corpus (repos/ fills 1:1 with
resolutions). 206 unseen tests + new repo-cache test pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The decision bank made extraction durable, but every restart still
re-paid the ~26-minute advisory enumeration + per-advisory resolution,
since those lived only in the local journal. Mirror them into the bucket
so a restart reads them back and goes essentially straight to any
remaining extraction.
bank.py: bank_resolution/resolved at <prefix>/<seed>/resolved/<sha256(ghsa)>.json
(streamed one-by-one as each advisory is decided), cache_advisories/
cached_advisories at <prefix>/<seed>/advisories/list.json, and
bank_repository/repositories (caches the advisory_index repo-resolution
pass, preserving renames and 404s). All gated by RESOLUTION_CONTRACT="v1"
(extract.py) -- bumped only when enumeration/eligibility/source.candidate
output changes; recent fetch/parse/pool changes do not bump it.
draft.py run(): with --bank, advisories load from the cache (else
enumerate then cache); eligibility decisions load from the bucket
(contract-matched) to skip re-resolution; new resolutions stream to the
bucket before the local checkpoint write. Repository lookups reuse the
repositories/ cache. No --bank => local-journal behavior unchanged.
Net: advisories + resolutions + extraction all resume from ousast-corpus;
restarts no longer re-enumerate or re-resolve. Fresh and resumed runs
produce byte-identical manifests at 1 and 4 workers. Bank methods proven
live against ousast-corpus. 356 tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The draw lost all progress on any restart because the local journal
invalidates on a code change and is the only cross-run state. Make the
bucket the source of truth: stream every per-repo accept/reject to
ousast-corpus as it is decided, keyed by sha256(repository+"\n"+fix) so
writes are idempotent (content dedup at rest), and resume by listing the
bucket and skipping already-decided repos. No supervisor/watchdog needed
-- a crash or a code change just re-runs and resumes.
New benchmarks/unseen/bank.py: corpus_settings (CORPUS_S3_* from .env,
missing key named, value never printed), CorpusObjects (lazy boto3,
path-style + SigV4 verified live against ousast-corpus @ 10.0.0.54:9001),
CorpusBank.bank/decided with one-retry put/get, split_decided. Decisions
live at <prefix>/<seed>/decisions/<key>.json with repository, fix,
accept/reject, reason, entry, exclusions, counts, contract, seed, ts.
draft.py: --bank [prefix] opens the bank on the coordinator (never shipped
to fork workers); run() resumes from bank.decided() before extracting,
reusing accepted entries across contracts (deterministic from repo+commit)
and rejections only when contract == EXTRACTION_CONTRACT ("v1", in
extract.py); collect() streams each decision one-by-one. Seed-order
acceptance, fork/used-set dedup, and duplicate-advisory reuse preserved.
Without --bank, local-journal behavior is unchanged.
CorpusBank proven live end-to-end (bank accept+reject, idempotent re-bank,
decided/split roundtrip). 325 tests pass offline, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The harvester cloned --filter=blob:none --no-checkout, then blame/log/diff
triggered the promisor to fetch each needed blob ON DEMAND -- hundreds of
round-trips per repo, the dominant cost that made the draw unable to
converge. The blobless clone was a premature optimization that backfired.
Drop --filter=blob:none from the clone and the two commit fetches. A
full-objects --no-checkout clone brings the whole commit graph AND every
blob in one pack, so blame/log/diff read locally; --no-checkout still
skips the working tree so disk stays bounded (plus the existing 500 MiB
clone_size_cap and the disk guard). With no promisor remote, a genuinely
missing object now errors loudly instead of silently round-tripping --
the instrument guarantee that the bulk fetch was complete.
Extraction semantics are unchanged (same SZZ introducing-commit positive,
same stratified ordinary negatives, byte-identical entries); only the
object-transfer strategy changes. Test inverted to assert no blob:none
filter on the clone; README updated. 145 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Git subprocess calls were deadline-bounded (Git.command_timeout uses the
remaining per-repo budget), but a pure-Python parse between git calls
(repository_changes -> history/blame/diff on a giant file) had no
deadline: when it hung, no further git call fired the git deadline, so
the worker stalled indefinitely (observed >720s with zero git
processes). An external watchdog was papering over it at ~20 min/stall.
Add extract.process_deadline(seconds): a SIGALRM/setitimer wall-clock
alarm installed on the worker's main thread that raises
ExtractionTimeout("wallclock_timeout"), with no-op guards (None/<=0, off
the main thread, or no SIGALRM support) and strict timer-cancel +
handler-restore in finally. Wrap the whole extract_repository body in it
so the timeout propagates to the existing `except source.Rejected` and
journals as a clean rejection, not a worker failure.
Budget = --extract-deadline (git, 240s) + --parse-deadline (new, 120s)
= 360s total ceiling, so git-bound repos keep their full budget (yield
unchanged) while a hung parse is rejected well before any external
watchdog. Record parse_deadline_seconds in the report.
295 unseen tests pass (incl. timeout + handler-restore + no-op cases),
ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The draw hardcoded 300 repos / 3 slices. At the measured lazy-fetch rate
the full 300 projects ~2.5 days; a 100-repo single slice (slice 1, all
the G0 baseline needs) is ~20h. Add --pool-size (default 300) and
--slices (default 3), validated (>=1, slices<=pool_size, evenly
divisible). Generalize assign_slices' stratified allocation from a
literal 3 to `slices`, the run() break / insufficient check / report
(changes = pool_size*13) to pool_size, and record both in the journal
inputs. Remove bisect's three-slice assumption; freeze reads the size
from the draft. Defaults (300/3) reproduce prior output exactly.
With --pool-size 100 --slices 1 the pool is the first 100 accepted repos
in seed order, all in slice 1 (deterministic, byte-identical across
worker counts). 280 unseen tests pass, ruff clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
