Commit Graph
604 Commits
Author SHA1 Message Date
Marius Ciepluch c237626f7a feat(access-control-prover): 9.2 monotone summaries over recursive call components 2026-10-29 18:30:18 +01:00
Marius Ciepluch fffa3e0002 feat(access-control-prover): 9.1 local function summaries 2026-10-29 18:25:23 +01:00
Marius Ciepluch 2466670ba3 feat(access-control-prover): 8.4 inheritance and base-declaration attachment through repo-defined descendants 2026-10-29 16:10:44 +01:00
Marius Ciepluch 8ef2629e0b feat(access-control-prover): 8.3 exact, finite-candidate and unresolved call targets 2026-10-29 16:04:51 +01:00
Marius Ciepluch 6f2154fae0 feat(access-control-prover): 8.2 stable declaration identities and per-change symbol index 2026-10-29 15:45:29 +01:00
Marius Ciepluch 9eee4d7f35 feat(access-control-prover): 8.1 capture call receiver and arity evidence in the semantic IR 2026-10-29 15:33:18 +01:00
Marius Ciepluch 03807f4ddd feat(access-control-prover): task 7.4 - judgment, trace and coverage contracts with abstract state 2026-10-29 15:23:43 +01:00
Marius Ciepluch 2f97fb2693 feat(access-control-prover): 7.3 authority abstract values and promoted implication facts 2026-10-29 15:06:53 +01:00
Marius Ciepluch 351583a04f feat(access-control-prover): 7.2 resource abstract values and operation-fact fields 2026-10-29 14:48:53 +01:00
Marius Ciepluch 5697a7fe9e feat(access-control-prover): 7.1 evidence-relation algebra and completeness value 2026-10-29 14:11:13 +01:00
Marius CiepluchandClaude Fable 5.1 03b114cbeb feat(prover): delivery D: control flow decides dominance; the trace carries each guard's relation, effect and the operation's control context
Opus agent delivery under spec access-control-prover, requirement 10. `semantic/cfg.py` builds a control-flow
graph per function from the syntax tree (if/else, switch arms, loops, try/catch/finally, return, throw, break,
continue, exit, assert) with dominators and a per-line control context, and leaves a function without a graph
rather than a wrong one. `obligations/cfg_dominance.py::CfgDominance` discharges a statement guard only when it
lies on every path to the operation and the operation is unreachable from its failing arm; a predicate whose
result nothing consumes is `value_only` and never discharges; router, decorator and earlier-hop witnesses behave
as before; without a graph the order rule applies and `cfg_unavailable` is recorded. The trace's guards carry
`relation` and `effect`, the trace carries `operation_context`; Java parameter lists carry names only; a
handler-named operation's arguments are the handler's parameters. The checker's default stays order-based.

Smoke on the 23 fix changes: 15 caught, 55 of 93 locations, precision 15 of 15, no failure, no change over
120 s (unchanged from delivery B). Gate: 3575 tests pass; the reserved-repository guard passes on the export.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 19:01:07 +02:00
Marius CiepluchandClaude Fable 5.1 2f628b921b feat(prover): delivery C: the ordered trace with argument provenance for every examined pair
Codex delivery under spec access-control-prover, requirement 9. `semantic/obligations/trace.py` serialises each
(entry, operation) pair the prover examined, discharged or unguarded: the entry with its boundary, access and
parameters; each hop in order with its call line, the guards before the call and the values forwarded; the
operation with the provenance of each argument traced through the function's binds; the guards after it; the
verdict. One canonical text form and one JSON form, no language, framework, repository, size or count fields.
The prover result, the arm record and each unguarded finding carry the traces. On the ec2-plugin fix: 32
traces, 4 unguarded. Gate: 3534 tests pass; the reserved-repository guard passes on the tracked export.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 18:22:45 +02:00
Marius CiepluchandClaude Fable 5.1 17e07b36c4 feat(prover): delivery B: handler-named operations, Symfony, Livewire, Remix and Tornado entries, enforcement witnesses, memoised path walk
Codex delivery under spec access-control-prover plus one fix of mine. Operation rows may now name a handler by
name pattern and framework (Jenkins form validation and list filling, Tornado request methods, Remix actions);
the mapper recognises Symfony, Livewire and Laravel controllers, Remix exports and Tornado handlers;
`obligations/enforcement.py` turns Laravel route middleware, Symfony attributes and security.yaml rules and
Tornado's authenticated decorator into router and decorator witnesses. The path walk looks each node's callers
up once per change and does not walk a tree with no recognised entry: one seatunnel change fell from 670 s to
35 s. One name Codex could not source (Tornado get_current_user) is left out and said so in the facts header.

Smoke on the 23 fix changes: 15 pushes caught (65 percent, Wilson 45 to 81), 55 of 93 locations, precision
15 of 15 flagged, no instrument failure, no change over 120 s. Delivery A caught 7.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 17:31:07 +02:00
Marius CiepluchandClaude Fable 5.1 beb272a5ec feat(prover): access-control prover, delivery A: path records, Stapler and Jetty entries, Java operation rows, the prover and its scoring arm
Codex delivery under spec access-control-prover. `semantic/obligations/paths.py` builds entry-to-function path
records from the cached caller index; `mapping.py` names Stapler web methods and Jetty handlers as entries;
`check_obligations` gains the opt-in premise `assume_entries_protected` with the label `unguarded_path`;
`hunt/prover.py` proves a change with no model call and states its coverage; `benchmarks/valid/prover_arm.py`
scores it with the valid-benchmark metrics. Two fixes after the first smoke run: a product file git holds empty
is read, not unread (four changes failed on `__init__.py` and empty assets), and absent-input reasons are
counted by name.

Smoke on the 23 fix changes of the training pool: 7 pushes caught (30 percent, Wilson 16 to 51), 28 of 93
locations, precision 7 of 8 flagged, no instrument failure; three changes over the 120 second bound. Twelve of
the 17 Jenkins fixes reach the fixed method but check no operation because the method is the operation;
the spec now carries handler-named operations for delivery B.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 16:50:04 +02:00
Marius CiepluchandClaude Fable 5.1 9feb8f1e33 fix(hunt): a row's entry points are the ones in its file or a caller's file, not the whole repository's list
The DSPy arm over the second pool failed with LMInvalidRequestError on every change with more than two rows, at no
cost: `source.facts_for` attached every entry point of the product tree to every row, 8190 records and 1.5 MB per
row on apache/seatunnel (the first pool's arm lost 11 changes the same way on trigger.dev and shopper), and the
rendered prompt was refused. The screen table of the second pool weighed 814 MB for the same reason.

A row now carries the entry points declared in its own file or in a file that calls the function, which is what the
input has described all along ("in-repository entry points that reach the file"). `source.entry_point_count`
follows that definition.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 14:47:08 +02:00
Marius CiepluchandClaude Fable 5.1 81ae206978 fix(hunt): build output gets no candidate rows, a minified line is not scanned once per function, the screen checkpoints every change
The second access-control screen spun for 2h20m on one ordinary mage-ai change: a Next.js build commit whose 3 MB
one-line bundles hold 12,355 functions each. `source.ranges_from_ir` ran two regex scans of the whole line once per
function, and the screen kept its 422 finished changes in memory only, so stopping it lost them.

- `source.ranges_from_ir`: a line over MINIFIED_LINE_CHARS (2000) is not checked per function; JavaScript names on it
  stay anonymous, other languages keep the parser's name.
- `source.generated_reason`: why a changed file is build output (a directory the product walk leaves out, a minified
  name, over MAX_FILE_BYTES, a line over the cap). `hunt.producers._candidates` leaves such files out and records the
  reason on the context; the fix site of a vulnerability is always kept. The DSPy arm shares the enumeration.
- The screen writes rows.jsonl and log.jsonl per change (rows first, entry last) and resumes by skipping logged ids.

Tests: the minified line never reaches the per-function scan; generated_reason on the four shapes and two source files;
build output in an ordinary change yields no rows while a vulnerability path in a one-line file is kept.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 13:24:45 +02:00
Marius CiepluchandClaude Fable 5.1 4cf9c85d22 perf(hunt): the caller search prepares each product file once per change
call_sites rescanned every line of every product file for declarations on each call; with one name per
candidate row that was the whole cost of the source producer. code_index prepares a file once (the names each
line declares), call_sites takes it, and callers_of keeps a cache of prepared files the producer context owns.
Same calls, tested against the unprepared scan. One Laravel change of 33 rows: 249 seconds before this pair of
commits, 20 now.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 09:46:46 +02:00
Marius CiepluchandClaude Fable 5.1 86d70831c8 perf(hunt): the source producer walks and reads the product tree once per change, not once per row
facts_for walked the whole product tree and re-read every product file for each candidate row; on a Laravel
application one change of 33 rows took four minutes, 96 percent of it there. The producer context already
holds the product file list and the texts it read for entry points, so facts_for and callers_of now accept
them. Same facts, tested by a call whose tree walk raises and whose file on disk is emptied after reading.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 09:35:16 +02:00
Marius CiepluchandClaude Fable 5.1 4063fd1562 feat(semantic): the hunter's semantic producer covers Java and PHP
Java parsed already, but the producer failed every Java row because no obligation facts existed for the
language and that failure took roles and def_use down with it. PHP had no grammar. Now: tree-sitter-php in the
semantic extra with PHP node kinds in the CST reader; obligation facts for Jenkins/Stapler, Spring Security and
Servlet code (java.toml) and for Laravel, WordPress and Symfony (php.toml), each header naming the training
clones and references the names were checked against; a language with an IR but no obligation facts keeps
roles and def_use and records obligations as unchecked (None), which the engineered features treat as missing,
never as zero. Proven on the committed pool: 6 of 6 Java rows and 33 of 33 PHP rows of one fix change each now
carry semantic features. Implemented by Codex; gated here with ruff, mypy and the full suite.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 09:35:16 +02:00
Marius CiepluchandClaude Fable 5.1 30ad197d3f fix(bench): an unparsable file in one ordinary commit no longer ends a repository's history walk
The combined access-control freeze stopped at its 31st repository: one ordinary commit of a PHP tree touched a
Python file that does not parse, functions() raised Rejected, and ordinary_history let it end the walk. The
file is now marked parse_failed on its row and such a commit is rejected as an ordinary push (reason
function_parse), which keeps the draw conservative without losing the repository.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 08:42:25 +02:00
Marius CiepluchandClaude Fable 5.1 351d73079a ci: the host job keeps the hunt and semantic extras; a benchmark main no longer leaks GIT_NO_LAZY_FETCH into later tests
The workflow runs pytest twice: in the dagger container (now one failure) and on the runner host, whose job
synced the dev group only, so dspy and tree-sitter were missing there (25 failures). The one failure shared by
both runs was the partial-clone extraction test: engine_trace.main sets GIT_NO_LAZY_FETCH in the process
environment, the variable outlived its tests, and git 2.44 and newer refuse the lazy fetch with "lazy fetching
disabled". The local git ignores the variable, so the leak was invisible here; the test module now restores
the variable after each test (leak checked with a session-end probe: present before, absent after).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 08:08:32 +02:00
Marius CiepluchandClaude Fable 5.1 17194b3df9 ci: pytest runs with the hunt and semantic extras kept; sandbox tests skip by cause where no bwrap exists
CI's first full pytest run since 2836b78 showed 65 failures in three classes. uv run re-synced the environment
to the lockfile's default extras before pytest, removing dspy again (UV_NO_SYNC=1 now). The learn slice, pair
corpus and hunt producer tests need tree-sitter (semantic extra installed). Thirty-two tests need bubblewrap
isolation, which CI's container cannot provide: they carry a sandbox marker and skip where the isolation
preflight fails, with the cause's type in the reason; proven locally by hiding bwrap (7 skipped, 23 passed).
The partial-clone extraction test now probes one lazy fetch through git itself first, so git's own stderr
reaches the next CI log instead of git_exit_128.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 07:54:09 +02:00
Marius CiepluchandClaude Fable 5.1 0d7d3f0dc9 chore: format the whole tree and lint the scratch measurement scripts; CI has been red since 25ae4b4
CI runs ruff format --check and ruff check over the whole repository; 29 files had drifted from the formatter and
four measurement scripts carried lint findings, so every push since 25ae4b4 failed before pytest ran.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 02:49:44 +02:00
Marius CiepluchandClaude Fable 5.1 dee23c23f5 measure: both arms over the access-control training pool (2 of 5 pushes, 4 of 48 locations each); runner books real cost
DSPy baseline: 2/5 pushes, 4/48 locations, 14 of 90 ordinary pushes flagged, 25 suspicions (3 true), 11 invalid-request
failures in shopper and trigger.dev, $6.13. Pilot hunter: 2/5 pushes, 4/48 locations, 99 of 100 ordinary pushes
flagged, 594 findings (3 true). Different pushes caught; nothing in trigger.dev.

The runner booked usd 0.0 per change so --budget-usd never stopped a run; it now prices every LM call of the change
at the registered prices and writes usd into the record.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 01:14:46 +02:00
Marius CiepluchandClaude Fable 5.1 fcb43fdd47 fix(hunt): an empty product file no longer fails the source producer on every row of a change
The 150 unread rows were not a checkout failure: the entry-point scan read every product file and one empty
file raised empty_source, which escaped the cached property and failed the source producer on every row of
the change (and re-read the whole tree per row). Empty product files are skipped; the row's own file still
has to be non-empty.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-14 00:53:37 +02:00
Marius CiepluchandClaude Fable 5.1 537b61e8d9 fix(hunt): diff producer records where a push removed lines; worktree checkout is proven complete
Two instrument findings from the first access-control feature table (952 rows, 74 positive rows):

1. A fix that adds a guard is, replayed base-to-head, a push that removes it. The diff producer only kept
   hunks with added lines at head, so 38 of 74 positive rows had no hunk in their file at all and 69 of 74
   were "not on a changed line", while 60% of ordinary rows were. The diff annotation separated the labels
   by construction and was blind to exactly the access-control shape. The producer (version 2) now records
   per file the head lines after which the push removed code; distance and on_changed_line count them, the
   model's hunks input renders them as removed_after_line, and diff.deletion_count is an engineered feature.

2. 150 rows (16%, every positive of one repository) read empty files from the worktree although git holds
   the blobs; the build ran while another process checked out from the same clone. _full_checkout now
   proves the checkout: git status must be clean, a dirty tree is restored once, then InstrumentFailure
   checkout_incomplete. The table script cross-checks every unread row against the blob size and refuses
   to screen a partly unread tree, and records the diff-scoped recall ceiling (positive rows a label-blind
   enumeration would reach).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 23:17:15 +02:00
Marius CiepluchandClaude Fable 5.1 3d08d692ff bench(train): class-targeted sourcing, --family, from the public OSV dumps
sources --family <family> keeps only advisories whose CWE the taxonomy maps to that family, writes
sources_<family>.json beside the training set, and excludes the pilot, every repository the
benchmarks already name, the strict used set and every earlier batch; --prefilter-only runs the
dump pass without GitHub. Implemented by Codex (gpt-6-astra), gated here: names normalize rejects
exclude nothing instead of aborting, and the strict used set is in the exclusions.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 21:46:45 +02:00
Marius CiepluchandClaude Fable 5.1 499f6c4474 fix(hunt): label every verified target of a change; undo an inherited sparse-checkout in the worktree
label_rows honoured only a change's primary site, so 43 of the 48 accepted access-control
locations were unlabelled; every entry of targets is now a positive with its own label_ref and
span. The training clones were emptied with sparse-checkout to save disk and a new worktree
inherits it: every file read failed and the source and semantic producers reported an unread tree
on all 599 rows. The runner now disables sparse-checkout in the worktree it adds.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 21:32:43 +02:00
Marius CiepluchandClaude Fable 5.1 1d1c87f63b bench(valid): a training freeze may cap the history walk at the newest N first-parent commits
ordinary_history(limit=) walks only the newest commits; assemble.freeze(history_limit=) and
--history-limit pass it through and the record carries it. The held-out benchmark passes none.
Reason: trigger.dev's 7,109-commit walk runs for hours at the per-commit cost of the function
and generated checks; the training freeze now walks its newest 1,500.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 20:47:23 +02:00
Marius CiepluchandClaude Fable 5.1 6f9613329e feat(hunt): candidates scored on folds through the runner; the pilot gain, once per accepted program (6.1)
rounds.py runs a candidate over compile or held-out folds through runner.run, so a tool-using
program has its tools bound to each change's checkout, and turns paired records into the
FoldResults accept.decide reads (closes the gap noted in bacd7a1). report.py runs an accepted
program once over the held-out pilot: a ledger line reserved before the run refuses a second run
per program id, contaminated or unchecked training runs are refused, the instrument is proved per
change and more than 5% failures makes the run invalid with no numbers quoted, the pilot's own
scorer and metrics give the after numbers, the recorded hunter numbers give the before, baseline
repeats pair with the single run, the proof arm's workload per 100 pushes is counted, and the gate
distance is read off the confidence curve. 124 hunt and stats tests pass. Implemented by Codex
(gpt-6-astra), gated here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 20:44:18 +02:00
Marius CiepluchandClaude Fable 5.1 bacd7a1e3f feat(hunt): proposals and the acceptance rule (5.1, 5.2)
propose.py: one change per round (instruction, demonstrations or one mask), examples built from
compile folds only with a provenance record of the change ids used, a registry for GEPA, MIPROv2
and BootstrapFewShot (MIPROv2 without the data-aware proposer so labels are never serialised),
compile-fold scoring with Wilson and the repository-paired bootstrap, and a compile-only history
the next round may read. accept.py: the rule as independent clauses (both folds non-negative with
one strictly positive, per-family regression budget on held-out recall, tie tolerance, no
acceptance without an interval, instruction cap, one change only), the comparison recorded in the
candidate artifact's configurations, rejections logged without held-out numbers. 48 fixture tests.
Known gap for live rounds: the optimisers and propose._score call the program directly, so a
tool-using program must be scored through the runner with its tools bound per checkout (6.1).
Implemented by Codex (gpt-6-astra), gated here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 20:22:22 +02:00
Marius CiepluchandClaude Fable 5.1 47d0f0fac1 bench(unseen): per-commit history cache for ordinary_history; assemble.freeze --history-cache
One append-only JSONL per clone (named by the sha256 of its top-level path), one line per commit
with the row as the walk built it and HISTORY_CACHE_VERSION, flushed after every commit so a
killed run keeps its progress. A cached row is reused only when its parents match the current log;
other versions and torn lines are recomputed. With no cache directory nothing changes. The
7,400-commit trigger.dev walk that took over an hour per freeze becomes a one-time cost.
Implemented by Codex (gpt-6-astra), gated here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 19:40:34 +02:00
Marius CiepluchandClaude Fable 5.1 dbaaa86d5b feat(hunt): contamination check (4.5): runs that used a reported repository are marked and kept out of any gain
contamination.py reads the repositories named by the held-out pilot manifest and by every
benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks
each run directory clean or contaminated with the report digest and time, and refuses to count
an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is
contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so
the package imports without the benchmarks tree.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 19:03:23 +02:00
Marius CiepluchandClaude Fable 5.1 6e5d060de6 feat(hunt): registered paired ablation, repository folds, leak audit, importance table and the keep/drop rule (4.2-4.4)
ablation.py registers the manifest (table digest, arms as masks, folds, metric, sample, budget,
rule, seed, program id) before any run and refuses a run or an analysis that does not match it.
Arms run over the same changes through runner.run, paired by change; effects on recall and on
suspicions per ordinary push are reported pooled and per family with the repository-clustered
bootstrap and exact McNemar, plus the recall-against-suspicions curve at every reported
confidence (never a tuned cut). The importance table and decisions read held-out effects only;
a neutral annotation survives only when a kept one needs it. 30 fixture tests with a scripted
model. Implemented by Codex (gpt-6-astra), gated here.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 18:49:58 +02:00
Marius CiepluchandClaude Fable 5.1 bbbb0f397e bench(valid): a training freeze may lower the ordinary floor to the ratio; the draw takes a size
The held-out benchmark keeps the 40-commit floor and the union of twelve-commit stratified draws.
A training freeze passes floor=20 (never below the ratio) and the record carries ordinary_floor.
When the union of twelve-commit draws cannot reach the count (small cells stay unallocated on
every draw), one stratified draw of the full count is used; with count equal to the eligible
commits that is every one of them.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 18:33:10 +02:00
Marius CiepluchandClaude Opus 5.5 2b65680d5d data: hunt training-set candidates (task 4.1), not frozen: 99 verified locations in 28 changes, awaiting spot-check
Sourced apart from the 12 pilot repositories: 212 candidates (pool-fresh, pool-retry, OSV for PyPI,
npm, Maven and Packagist), 49 dropped because the benchmarks already name their repository, 413
catalog locations, 99 passing all three verifier checks in 28 repositories. Short of the 40-positive
target. catalog.py now reads .java and .php. Freeze waits on the maintainer's spot-check
(CANDIDATES.md, SOURCING.md).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-13 16:48:41 +02:00
Marius CiepluchandClaude Opus 5.5 25ae4b489c fix(p1,p0): restore the pre-P1 range contract for two callers; docs link and maintainer paths
The full suite after the replay found regressions from 062a7b7 and 94e96c6:
- catalog_gen and candidates.labeled_spans still unpacked 3-tuples and expected None on a parser
  rejection; source.ranges returns FunctionRange (4 fields, module entry first) and raises ValueError.
- ranges_from_ir passed the language to declarator_name, so the repo-facts declaration table named a
  callback call (exec(..., function () {) as a function; the pre-P1 call had no language.
- the catalog_gen fail-loud test blocked the old module name.
- docs: a renumbered examples anchor, and two .kiro paths in the public threat model.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-13 16:15:32 +02:00
Marius CiepluchandClaude Opus 5.5 455637bea5 fix(hunt): a malformed tool-loop step ends the loop and the answer is still extracted
DSPy 3.4.1 raises AdapterParseError for an unparseable ReAct step, which is not the ValueError its loop
catches, so one malformed step discarded the whole hunt (7 of 60 hunts in the repeats). The step now ends
the loop like an invalid tool choice. Records carry loop_parse_failed beside parse_failed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-13 15:30:13 +02:00
Marius CiepluchandClaude Opus 5.5 602ec23e03 fix(hunt): lines read through a tool count as shown; an unparseable answer is an empty answer
The answer check counted only the rendered function text as shown, so 7 of
8 baseline suspicions citing lines the model had read with read_file were
marked unknown (baseline 1/12 against the pilot hunter's 6/12). Tools now
record the lines they return. A DSPy parse failure scores as zero findings
with a flag instead of a broken run. The per-run response cache now takes
effect through dspy.configure_cache.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-13 13:10:40 +02:00
Marius CiepluchandClaude Opus 5.5 c5bf52f47d measure: Semgrep arm on the valid pilot, our rules and facts converted
0 of 12 positives; 12.6 false alarms per 100 ordinary pushes (2.1 on changed
lines only). The misses come from what the rules and facts express (access
control lives in obligations, which do not convert; sources cover request
objects only), not from the engine. Comparison arm only: a Semgrep miss is
never evidence against a hunter finding.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-13 13:10:19 +02:00
Marius CiepluchandClaude Fable 5.1 c95b9241d8 feat(stats): one statistics module on scipy, statsmodels and scikit-learn; engineered features tied to the annotations
openultrasast.stats: Wilson intervals (statsmodels), exact McNemar, a
repository-clustered percentile bootstrap and its paired form with one
quantile convention, grouped folds (GroupKFold, leave-one-group-out), and
per-column separation (ROC AUC) for the leak audit and the signal screen.
Checked against the benchmark scorer's Wilson and the learn harness's
McNemar. hunt/features.py turns the feature table into the engineered
frame (distances, counts, flags) and keeps the map from every engineered
feature to the annotation the model is shown, so an ablation mask drops
both, and screen() says which annotation carries signal before a model
call is spent. The [hunt] extra now carries the libraries; the core
install is unchanged. The learn harness's own copies stay until that line
is revived.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 12:43:02 +02:00
Marius CiepluchandClaude Fable 5.1 d9d6335ab3 feat(hunt): feature table, producers, labels, metric, artifact, the DSPy hunt program, runner and CLI
hunter-learning-loop groups 1-3 (first increment). The feature table is the
contract: one row per touched function with diff, source, semantic and
taxonomy columns, provenance per producer, labels by the benchmark's rule.
Producers read the repository at head only (a planted advisory id and
commit text never reach a feature). The hunt program is a DSPy ReAct loop
over a signature whose inputs are the unmasked columns in a fixed order;
answers outside the closed family list or citing unseen lines score as
unknown. The runner writes records the benchmark scorer reads, resumable
by change, program id and mask, and refuses the held-out pilot unless the
run is a report run. Research extra [hunt] = dspy, pandas; the core
install is unchanged. 43 tests on a scripted model, no network.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 12:33:58 +02:00
Marius CiepluchandClaude Fable 5.1 062a7b780e refactor(p1): one source-facts module owns function ranges, callers, declaration tables and language aliases
src/openultrasast/source.py replaces the copies in semantic/functions.py,
plane/tasks/repo_facts.py, learn/features.py, learn/excerpt.py and
plane/tasks/verify.py. Two engines share one sentinel (<module>), one
DECLARATION/KEYWORDS/RESERVED set and one alias map: the parser engine
(ranges, enclosing, bounds, call_sites, callers_of, facts_for) for the
scorer, the overlay and the hunter's annotations; the line-pattern engine
(declared_functions, enclosing_at, function_span, function_bounds, the brace
matcher) that labels, features, roles, excerpts and plane facts were built
on. learn no longer imports plane.tasks. Frozen records that store the old
sentinel are mapped on read (module_function).

Gate: labels rebuilt before and after are identical, 7,921 rows, after
sentinel normalisation (benchmarks/measurements/2026-10-13-source-module-
label-diff/record.json). 490 targeted tests pass; 3,312 collect. Net: 657
lines deleted, 212 added. benchmarks/unseen keeps <global> for its own
frozen manifests.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 12:00:37 +02:00
Marius CiepluchandClaude Fable 5.1 f63703a604 chore(p0): drop never-enabled hunter tools, test-only functions and receiver delivery
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: the four hunter tools no caller enables (read_definition,
entry_points, flows, obligations) with their schemas and dispatch, the dead
mapping branch of find_refs; 20 functions whose only caller was one test
(mapping task records, compute_ranking_metrics, build_quick_hunter_prompt,
validate_evidence_transition, should_fuse, harness processors,
rank_targets_with_model, model/report slice rendering, taxonomy
hierarchical/related, config_value.verdict, taint.verdict, roles.load_roles,
features.language_of and build_for_scan, folds.leave_one_*,
features.recorded_case, search BatchExecutor and the three __main__ shims,
delivery.put_input); the receiver delivery mode of the plane. push_inputs.py
moved to benchmarks/push. memory.purge stays: the live-S3 test cleanup uses
it. Net: 2,730 lines deleted, 207 added. 473 targeted tests passed; 3,282
tests collect. breadth.toml records the removals.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 11:06:09 +02:00
Marius CiepluchandClaude Fable 5.1 f5a904b7f2 chore(p0): retire regress, the skill router, the module audit tool and vocabulary.sc
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep
now records a regress_retired degradation and equals standard; proof is the
search-with-proof arm's job), the sandbox docker runner and probe that only
regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers
only regress called; skills.py and index.py with the hunter pool's skill
routing; model/audit.py; cpg/queries/vocabulary.sc with
benchmarks/push/ontology_gap.py. Docs and the scan skill describe the
retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed,
28 skipped; 3,331 tests collect.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 10:50:01 +02:00
Marius CiepluchandClaude Fable 5.1 94e96c6885 chore(p0): remove the improve lever and the plane improvement loop
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/improve/ (1,175 lines), the ousast improve
subcommand, plane/tasks/loop.py and loop.yaml with the --loop generation,
the memory-snapshot step in memory.seed, pairs.build_pair_signals, and the
four improve tests plus the loop cases in other tests. Docs no longer
describe the lever. Net: 3,073 lines deleted, 83 added. Targeted tests:
154 passed; 3,436 tests collect.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 10:34:57 +02:00
Marius CiepluchandClaude Fable 5.1 23aeeada10 feat(benchmark): combine --keep-verdicts drops hunter suspicions the classifier rejected
Opt-in filter on the classifier verdict for hunter-sourced defects; the default
keeps everything. Each combined record notes the kept verdicts and the counts.
Live check on the pilot: (a)+(b) filtered to 'vulnerable' gives recall 5/12 and
54.4 false alarms per 100 ordinary pushes, matching the scratch numbers.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-13 08:21:59 +02:00
Marius CiepluchandClaude Fable 5.1 abd82c6e33 fix(benchmark): meter the provider client before building the classifier Caller
Live run: every classification failed with ValueError because Caller.ask
reads client.usage as a dict (the learn CLI always hands it a MeteredClient)
while a bare DeepSeekChatClient keeps usage as a list of rows. build_caller
now wraps a client that carries a usage list in MeteredClient; scripted and
already-metered clients pass through unchanged.

Regression test runs classify_family through the real DeepSeekChatClient
wrapper with a stub transport (the module fixture that stubs classify is
restored for it). The error field stays the exception type only, since a
message can carry a model key.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-12 20:47:12 +02:00
Marius CiepluchandClaude Fable 5.1 d37d9634c6 fix(benchmark): classify hunter findings into the taxonomy before scoring arm (b)
The tool hunter's family tag is the model's free text (a live run returned
family:sql-parser and family:readonly-enforcement), so every arm (b) defect
scored family "unknown" and score.catches() could never credit the hunter
even when it landed on the labelled file.

hunter_arm now classifies each finding into the closed taxonomy vocabulary
with the decision-engine classifier (learn.program.classify with
families=FAMILY_IDS minus unknown): a numbered, clamped excerpt around the
finding sets valid_lines, the canonical ANSWER_FORMAT is reused, an off-list
family parses to "none" and lands as "unknown". The benchmark label is never
a fallback. The defect keeps the raw hunter family, title and rationale plus
the classifier's verdict, confidence, rationale, cited lines and parse
failures for audit; the record counts classifier calls and families.
--no-classify keeps the old heuristic path.

14 new offline tests, 67 green with the benchmark tests.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-12 20:42:08 +02:00
Marius CiepluchandClaude Fable 5.1 9f7ca132ff feat(benchmark): hunter arm runner and combiner (group 6)
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.

`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.

9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
2026-10-12 20:29:03 +02:00