Codex delivery under spec access-control-prover plus one fix of mine. Operation rows may now name a handler by
name pattern and framework (Jenkins form validation and list filling, Tornado request methods, Remix actions);
the mapper recognises Symfony, Livewire and Laravel controllers, Remix exports and Tornado handlers;
`obligations/enforcement.py` turns Laravel route middleware, Symfony attributes and security.yaml rules and
Tornado's authenticated decorator into router and decorator witnesses. The path walk looks each node's callers
up once per change and does not walk a tree with no recognised entry: one seatunnel change fell from 670 s to
35 s. One name Codex could not source (Tornado get_current_user) is left out and said so in the facts header.
Smoke on the 23 fix changes: 15 pushes caught (65 percent, Wilson 45 to 81), 55 of 93 locations, precision
15 of 15 flagged, no instrument failure, no change over 120 s. Delivery A caught 7.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codex delivery under spec access-control-prover. `semantic/obligations/paths.py` builds entry-to-function path
records from the cached caller index; `mapping.py` names Stapler web methods and Jetty handlers as entries;
`check_obligations` gains the opt-in premise `assume_entries_protected` with the label `unguarded_path`;
`hunt/prover.py` proves a change with no model call and states its coverage; `benchmarks/valid/prover_arm.py`
scores it with the valid-benchmark metrics. Two fixes after the first smoke run: a product file git holds empty
is read, not unread (four changes failed on `__init__.py` and empty assets), and absent-input reasons are
counted by name.
Smoke on the 23 fix changes of the training pool: 7 pushes caught (30 percent, Wilson 16 to 51), 28 of 93
locations, precision 7 of 8 flagged, no instrument failure; three changes over the 120 second bound. Twelve of
the 17 Jenkins fixes reach the fixed method but check no operation because the method is the operation;
the spec now carries handler-named operations for delivery B.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Codex stopped on a real conflict: with path records and no clause or sibling norm the checker returns no finding
by design. The prover's premise (a recognised entry is protected unless declared public) is now an opt-in keyword
with its own label and evidence, default off, existing behaviour and tests unchanged.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A deterministic check that a protected operation is reachable from an entry point without a check dominating
the path, per framework, built on the existing obligations checker, caller index and entry mapper; scored as a
rule arm on the access-control training pool. Follows the maintainer's finding that the hunter's features
describe shape and cannot decide unsafety.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
rounds.py runs a candidate over compile or held-out folds through runner.run, so a tool-using
program has its tools bound to each change's checkout, and turns paired records into the
FoldResults accept.decide reads (closes the gap noted in bacd7a1). report.py runs an accepted
program once over the held-out pilot: a ledger line reserved before the run refuses a second run
per program id, contaminated or unchecked training runs are refused, the instrument is proved per
change and more than 5% failures makes the run invalid with no numbers quoted, the pilot's own
scorer and metrics give the after numbers, the recorded hunter numbers give the before, baseline
repeats pair with the single run, the proof arm's workload per 100 pushes is counted, and the gate
distance is read off the confidence curve. 124 hunt and stats tests pass. Implemented by Codex
(gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
propose.py: one change per round (instruction, demonstrations or one mask), examples built from
compile folds only with a provenance record of the change ids used, a registry for GEPA, MIPROv2
and BootstrapFewShot (MIPROv2 without the data-aware proposer so labels are never serialised),
compile-fold scoring with Wilson and the repository-paired bootstrap, and a compile-only history
the next round may read. accept.py: the rule as independent clauses (both folds non-negative with
one strictly positive, per-family regression budget on held-out recall, tie tolerance, no
acceptance without an interval, instruction cap, one change only), the comparison recorded in the
candidate artifact's configurations, rejections logged without held-out numbers. 48 fixture tests.
Known gap for live rounds: the optimisers and propose._score call the program directly, so a
tool-using program must be scored through the runner with its tools bound per checkout (6.1).
Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
contamination.py reads the repositories named by the held-out pilot manifest and by every
benchmark report (accepted_repos.json and NOTES.md links under benchmarks/measurements), marks
each run directory clean or contaminated with the report digest and time, and refuses to count
an unchecked run toward a gain. ablation.analyse carries excluded_from_gain when its run is
contaminated. Implemented by Codex (gpt-6-astra), gated here; the benchmark import is lazy so
the package imports without the benchmarks tree.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
ablation.py registers the manifest (table digest, arms as masks, folds, metric, sample, budget,
rule, seed, program id) before any run and refuses a run or an analysis that does not match it.
Arms run over the same changes through runner.run, paired by change; effects on recall and on
suspicions per ordinary push are reported pooled and per family with the repository-clustered
bootstrap and exact McNemar, plus the recall-against-suspicions curve at every reported
confidence (never a tuned cut). The importance table and decisions read held-out effects only;
a neutral annotation survives only when a kept one needs it. 30 fixture tests with a scripted
model. Implemented by Codex (gpt-6-astra), gated here.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
A Semgrep, quick-rule or engine match may enter the feature table as
positive evidence; its absence is never a feature, filter, demotion or
gate on a hunter suspicion, and no acceptance rule counts it. The Semgrep
arm is a comparison only and may at most replace the regex quick tier.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Engine LGPL-2.1, shelling out carries no obligation; the registry rules are
under the Semgrep Rules License since 2024-12-13 and cannot ship, only our
own converted rules can. Measured 4.7 s startup and 5 to 10 ms per file;
converted regex and taint rules fire for Python, PHP and JavaScript; C
propagation and Groovy do not; one sanitizer semantic we refuse on purpose
is accepted by Semgrep. Recommendation: a Semgrep arm on the valid pilot
first. Recorded in the hunter-learning-loop research note and breadth.toml.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
610 pair files, 570 labelled functions. lizard matches the parser engine on
C, Python and Java but loses JavaScript names and TypeScript end lines:
keep ours, take its cyclomatic complexity as a feature candidate.
tree-sitter-language-pack gives identical spans, is faster, replaces six
grammar wheels with one and adds Go, Ruby, PHP and Perl: adopt as grammar
packaging. Also found: .c files should try the c grammar before cpp.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Six groups: table and producers; the hunter as a program with the search
metric and the run artifact; runner integration and a baseline on the 12
pilot positives; training set, folds, paired ablation and the importance
table; proposals and the acceptance rule; the pilot gain report.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Feature engineering is the centre: a feature table with switchable columns
is the contract between producers (source.facts_for, semantic facts, entry
points, diff hunks; lizard and Semgrep only after their evaluations), the
ablation runner and the hunter program. DSPy runs the program (ReAct over a
HuntSignature; GEPA and MIPROv2 propose), our rule accepts on held-out
folds. The pilot stays held out; the hook, its client and the core install
are untouched. Requirements approved by the maintainer ('4 first').
Co-Authored-By: Claude Fable 5.1 <[email protected]>
src/openultrasast/source.py replaces the copies in semantic/functions.py,
plane/tasks/repo_facts.py, learn/features.py, learn/excerpt.py and
plane/tasks/verify.py. Two engines share one sentinel (<module>), one
DECLARATION/KEYWORDS/RESERVED set and one alias map: the parser engine
(ranges, enclosing, bounds, call_sites, callers_of, facts_for) for the
scorer, the overlay and the hunter's annotations; the line-pattern engine
(declared_functions, enclosing_at, function_span, function_bounds, the brace
matcher) that labels, features, roles, excerpts and plane facts were built
on. learn no longer imports plane.tasks. Frozen records that store the old
sentinel are mapped on read (module_function).
Gate: labels rebuilt before and after are identical, 7,921 rows, after
sentinel normalisation (benchmarks/measurements/2026-10-13-source-module-
label-diff/record.json). 490 targeted tests pass; 3,312 collect. Net: 657
lines deleted, 212 added. benchmarks/unseen keeps <global> for its own
frozen manifests.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: the four hunter tools no caller enables (read_definition,
entry_points, flows, obligations) with their schemas and dispatch, the dead
mapping branch of find_refs; 20 functions whose only caller was one test
(mapping task records, compute_ranking_metrics, build_quick_hunter_prompt,
validate_evidence_transition, should_fuse, harness processors,
rank_targets_with_model, model/report slice rendering, taxonomy
hierarchical/related, config_value.verdict, taint.verdict, roles.load_roles,
features.language_of and build_for_scan, folds.leave_one_*,
features.recorded_case, search BatchExecutor and the three __main__ shims,
delivery.put_input); the receiver delivery mode of the plane. push_inputs.py
moved to benchmarks/push. memory.purge stays: the live-S3 test cleanup uses
it. Net: 2,730 lines deleted, 207 added. 473 targeted tests passed; 3,282
tests collect. breadth.toml records the removals.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Off every active line (PATHFINDER-2026-10-13/03-unified-proposal.md, cuts).
Deleted: src/openultrasast/regress/ and the deep-mode REGRESS branch (deep
now records a regress_retired degradation and equals standard; proof is the
search-with-proof arm's job), the sandbox docker runner and probe that only
regress used, --fail-on worth-fixing, RegressConfig, the two ledger helpers
only regress called; skills.py and index.py with the hunter pool's skill
routing; model/audit.py; cpg/queries/vocabulary.sc with
benchmarks/push/ontology_gap.py. Docs and the scan skill describe the
retirement. Net: 3,895 lines deleted, 86 added. Targeted tests: 180 passed,
28 skipped; 3,331 tests collect.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
PATHFINDER-2026-10-13/: feature inventory, ten per-unit flowcharts with
file:line evidence, cross-feature and within-unit duplication reports, a
proposal that keeps every package on one of the four active lines, and
copy-ready plan prompts in order. .kiro/breadth.toml: every spec and
capability with a status and the deletion that closes it. No code changed.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
fossil-mcp dead-code over src found 72 candidates; 69 are referenced by tests,
__all__ or other modules, 3 are not and are removed (cli._first_source,
pairs._scorable, provider.openrouter._extract_message_content). Eight spec
directories that steering already called retired or superseded leave the tree
(history keeps them, commits named in overview.md): harnessx-removal,
harnessx-self-improving-rulesets, openrouter-sast-harness, learning-harness,
constrained-detector, three-stage-scan, real-world-vfc-slice,
unseen-repo-evaluation. overview.md gets a current-state section: the valid
benchmark numbers, what they say, and the four active lines in order.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The hunter gets an annotation model (named, versioned facts about the push)
and statistical ablation decides which annotations stay. Labels from fix
sites and ordinary pushes, training set apart from the held-out pilot,
compiled replayable program, train-and-holdout acceptance rule, two stages
reported separately, no thresholds set in this spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
benchmarks/valid/hunter_arm.py runs the shipped deep-mode tool hunter
(tool_hunter.run_tool_hunter via resolve_hunter_client) over each frozen
change off-cluster: detached worktree at head with git hooks disabled,
hotspots scoped to the changed files, instrument rule on every hotspot
(bytes read printed, empty or unreadable fails the change loudly), model id
from the config resolver, family from the taxonomy, never from the label.
Records land in the exact shape score.py reads (push.admission.defects with
locations), resumable per change.
`combine` joins arm (a) replay records with arm (b) hunter records per change
id, dedupes defects by (path, line, family), keeps (a)'s instrument, records
which arms were present, and writes a file score.py loads directly.
9 offline tests (stub hunter, real scorer), 53 green with the benchmark
tests. No network, no key handling, no live run.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
The pre-push hook has no LLM detection tier today: push/assistance.py only
selects a witness for already-admitted defects and cannot add a finding. The
hunter lives in scan --mode deep. The maintainer's call is that the LLM tier is
required on pre-push, so R6 now measures two arms over the same changes:
(a) the hook as shipped (quick rules + engine), (b) the hook plus the hunter
arm run off-cluster. (b) is the number the tool is held to; the gap is product
work in pre-push-safety-net, not a benchmark artefact.
Adds group 6 (hunter arm runner + combine + offline test) to tasks, aligns the
brief and design, and labels the overnight replay as arm (a).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Builds the benchmark from .kiro/specs/valid-detection-benchmark/brief.md:
sound labels from the fix (no SZZ), app-shaped verified positives,
ordinary negatives from the same repos, whole tool on unseen repos,
standard metrics.
Spec: requirements.md, design.md, tasks.md. Reuses the existing sourcing,
fix-site labelling, negative drawing, replay, and Wilson scoring; drops
the SZZ layer.
New offline modules under benchmarks/valid/:
- catalog.py positives labelled at the fix site (never introducing/last_touch)
- verify.py one LLM call per case (vulnerable? fix addresses it? reachable
in-repo?) + human spot-check; this is the app-shaped filter.
Honors the instrument rule: bytes read per function, reject on
an empty read.
- assemble.py >=20x ordinary negatives + a frozen, deterministic manifest with
the held-out guard
- metrics.py precision (PPV), false alarms per 100 pushes, exploit-verified
rate, and the plain headline, layered on score.py
36 new offline tests, 495 regression tests green. No network, no harvest,
no model calls in tests. Group 5 (the live pilot) is not started.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lead with the names a security engineer already knows instead of the
coined ones:
- catch rate -> recall (true-positive rate)
- false-alarm rate -> false-positive rate (still reported as false
alarms per 100 pushes)
- proved-catch / proof rate -> exploit-verified findings (a working
proof of concept)
- precision stays precision (positive predictive value)
The plain-English question for each metric stays as the explanation.
Updated the valid-benchmark brief and the roadmap. No numbers changed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran the humanizer rules over 79 prose files in the active specs (brief,
design, requirements, research and narrative docs). Removed em dashes,
significance inflation, forced rule-of-three, bold-label list items and
filler. Skipped tasks.md checklists and the retired specs (harnessx-*,
openrouter-sast-harness, unseen-repo-evaluation).
Every number, path, command, hash and identifier was preserved: checked
each file against HEAD for drift in numbers, inline code and code fences,
zero drift across all 79 files.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The frozen SZZ-labeled pool is not a valid benchmark. SZZ puts the vulnerability in
the wrong place about half the time, and ~76% of entries are library-internal (not
reachable in-repo), so its catch/false-alarm numbers (record 07's "3/92") are noise.
Retired as a metric and a gate across the roadmap and briefs; no pool-scored number
is quoted anywhere.
Added .kiro/specs/valid-detection-benchmark/brief.md: positives labelled from the
fix (not SZZ), kept only when app-shaped and reachable in-repo, each verified;
ordinary negatives from the same repos; unseen repos; the whole tool measured; plain
metrics (catch rate, false-alarms per 100 pushes, precision, proved-catch).
Ran a plain-English pass (the humanizer rules) on the roadmap, README, and the two
active briefs via Codex, then reviewed the diff: only these files changed, every
number/path/hash/command is unchanged, and the retired-metric decision is intact.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roadmap "Where we are" table said Baseline (G0.2) = "Not yet, run on kube" -- stale.
Record 07 (G0 exit) was completed right after on the docker lane (e901640, 1300/1300,
catch 3.3% / FP 3.1%, regex/quick-rule floor, engine_covered=0). G0 is exited; the kube
`ax` lane was never needed for the baseline. Reframed the readiness plan: steps 1-4 done,
the `ax` lane moves to G1 (engine arm at scale, gated by spec tasks 4.4/5.4).
Also fixed the ax binary in the coordinator commands: use $HOME/go/bin/ax, NOT
$HOME/.local/ax-v0.3.1/ax (the latter hits the AX-lane dispatch blocker recorded in the
unseen spec tasks.md). README.md + benchmarks/unseen/README.md.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Call-graph trace: no shipped product path or in-repo benchmark/test reached the
real single-query JoernBackend.query("taint"). The push/scan/replay path is
double-shielded -- batch fetch (run_batch) in Phase 2, and a _prefetched replay
CpgResult in Phase 3 whose run ignores the query -- so scan_region's cpg.run
("taint") never reaches JoernBackend.query. Benchmarks use the batch path; tests
use fake canned-row results. Only out-of-product ad-hoc harnesses hit it.
Regression onset 562b670 (2026-09-25).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
taint.request_params emits guards/fixedOrigin/originAnchors/quotedSanitizers/
fieldParameterSources, which taint.sc reads from per-request JSON in its BATCH
branch but does NOT declare on `@main def exec` and does NOT consume in its
single-request branch (it defaults them). The single-query path passed them as
`--param guards=...` (disposable) or named args (session), so joern errored
`Unknown arguments` and every cpg.run("taint", ...) failed -- a silent zero of
the shape AGENTS.md warns about. scan_region->taint_verdict->_usable_flows and
the pair-path verdict()/verdicts() were dead; the push/scan batch route was
unaffected (it carries the extras as JSON the script parses itself).
Fix: parse the script's `@main def exec(...)` signature once (_declared_exec_params
+ memoised _declared_params) and, in query(), filter params to the declared set
before building either route. Faithful, not lossy -- the single-request branch
could never use those 5 fields. No-op for config/dominance (their request_params
emit only declared params; verified). Only taint sheds the 5 extras.
Verified end-to-end on banks (patched backend in the replay image): single-query
path now runs; baseline 0 findings, annotated search 2 path/model_entailed
findings (prompt->write_text). Regression test asserts no undeclared param
reaches the command line and the query parses, with a fake runner (no Joern).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Same CPG, only the search changes. Baseline (shipped path spec) resolves 0
prompt->write_text flows on banks; injecting the two annotations an LLM steer
would supply (parameter_sources=True + write_text/read_text path sinks) makes the
engine resolve prompt -> path-join -> prompt_file.write_text(prompt.raw). First
measured instance of "#2 guided by #1" closing on an unseen target; fix needed
exactly the param-source + file-write-sink annotations.
Files the instrument bug the first MVP hit: single-query cpg.run("taint",
request_params(...)) is dead -- request_params emits 5 keys (guards/fixedOrigin/
originAnchors/quotedSanitizers/fieldParameterSources) absent from taint.sc's exec
signature; both single-query sub-routes reject them (silent None). Batch path
(run_batch->requestsFile JSON) unaffected; push/scan route uses batch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 hits have a runtime oracle (search FAMILY_ORACLES: injection->sql/command, ssrf for
untrusted_destination -- corrects the earlier families.toml-based 'dcmcp unprovable'). But
search/verify.verify() needs an AUTHORED runnable demo + both head/base building OFFLINE + a
sandbox differential (3/3 affected vs 0/3 safe). The scan/push path reaches only suspicion/
static_corroboration; no code path from a scan suspicion to exploit_demonstrated, no Coordinator
caller from the push flow. So proof-arm lift AT PUSH TIME = 0/3 -- proving is a per-target,
offline-build, runtime, differential operation that can't run inside a pre-push hook (prior
per-pair demos were mostly inconclusive on this wiring).
Strategic consequence: proof/BLOCK is an ASYNC lane (CI/plane/triage with authored demos), not
pre-push. Honest pre-push product = advisory-with-conditions (bet #2); proof arm = async BLOCK
lane (bet #3). Local sandbox exists (userns).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Ran ousast scan --mode deep on the 5 app-shaped + sound pool targets (the fair change-scoped
eval set from the shape classification). Detection 3/5: graphiti (hunter, Cypher injection),
mcp-remote (hunter, url-scheme injection), dcmcp (ENGINE overlay -- 'outbound.request reached
from parameter' on readFileFromUrl; parameter-as-source fired because an entry point is present).
2/5 missed: malla (DOM/template XSS) and dms (Jinja SSTI), both in families families.toml
documents as limited.
This validates the strategic pivot: the near-zero catch was substantially a TARGET MISMATCH --
on app-shaped repos the engine itself detects (0 -> 3/5). But all hits stop at suspicion/
static_corroboration; release-grade BLOCK still needs proof-arm elevation + family coverage +
the DSPy hint loop. Cost ~$0.007/repo. So bet #1 (fix eval target) is necessary and material
but not sufficient.
Next: recompute a floor on an app-shaped sound subset / real app corpus (VAmPI, NodeGoat,
DjanGoat, vibe-py); measure proof-arm suspicion->proven lift.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Named the core problem with the maintainer: the tool spots dangerous ops but cannot prove
attacker-reachability -- the trust boundary lives outside the changed library code. Chosen
strategic bet (of three): FIX THE EVALUATION TARGET FIRST.
LLM shape classification of the 49 public pool vulns ($0.033): reachability_in_repo only
24% (12/49); 76% are pure libraries. A legitimate change-scoped catch needs sound label AND
reachability in-repo -> just 5/49 (~10%): graphiti, desktopcommandermcp, malla,
document-merge-service, mcp-remote. Shape predicted the hunter outcome (graphiti app-shaped ->
caught; bedrock library -> missed). So record-07's 3/92 is scored against a ~90%-illegitimate
denominator -- the near-zero catch is largely a target mismatch.
Next: deep scan on the 5 fair targets for a per-tier catch rate; then an app-shaped eval set
from known-vulnerable apps. Artifacts: 2026-10-12-pool-p1-shape-classification/ (+ earlier
label-audit, hunter-sound-targets). Memory: evaluation-target-mismatch.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
ousast scan --mode deep (DeepSeek deepseek-flash, model layer + tool_hunter, 60-call bound) on
bedrock + graphiti, the two sound pilot targets the static engine missed:
- graphiti: HIT -- falkordb_driver.py:207, tag family:cypher-injection; rationale traces
build_fulltext_query concatenating user input into a Cypher string at line 278 (inside the
labelled function, location-matches). The static engine produced zero findings here.
- bedrock: miss -- 12 suspicions on adjacent concerns (network-PUBLIC, download_file), never
named the install_packages -> pip command injection.
Cost is a non-issue: ~$0.007/repo, 64 model calls for a whole-repo deep scan. BUT everything
stopped at suspicion rung -- the proof arm did not elevate to proven; release-grade BLOCK needs
proven findings.
Direction (maintainer): prompt evolution is a core lever -- a DSPy-based loop (GEPA/MIPRO,
pluggable; evolving field) whose objective is to learn what hints/annotations steer the model
to the real sink, sourced from the framework knowledge already shipped (no new user
annotation), with LLM-proposed boundaries graduating into the facts.
Next: measure proof-arm lift on graphiti; stand up the DSPy hint loop + re-run bedrock; widen
the hunter to the 23 sound targets for a catch rate; then a change-scoped + proven run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Widened the banks label-mislocation finding across all 49 public pool vulns via DeepSeek
(advisory + head code + pool label -> sound/unsound), $0.038 total. Result: 23/49 (47%)
sound; 53% unsound (25 pre-exist in base, 16 sink outside the changed lines) -- a change-scoped
hook correctly stays quiet there. Systematic by family: injection 10/13 sound, but path 1/7,
output_encoding 1/8, config_secrets 1/4.
Implications: record-07's floor (catch 3/92) is scored against a corpus ~half unfair to a delta
hook; the honest catchable denominator is the sound subset (caveat added to record 07 NOTES).
Fair pilot re-score: only bedrock + graphiti (injection) of the 5 are sound and the engine
missed both = the real detection gaps for the LLM-guided hunter.
Also: GEPA-style reflective-retry (future spec) + GLM cyber model (future spec) noted; DeepSeek
is the working default. Artifact: benchmarks/measurements/2026-10-12-pool-p1-label-audit/.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
DeepSeek deepseek-flash reachable: ~$0.0002/call, ~1.1s (sub-cent/push). The LLM gives the
correct verdict WHEN aimed at the real sink + source context: asked about the pool's
SZZ-labelled site (_load) it correctly says not-vulnerable; asked about the real sink
(from_prompt_path line 44, prompt.name request-derived) it correctly says vulnerable. So '2
guided by 1' works as a mechanism.
Two blockers, both concrete: (1) it needs the agentic search-with-proof hunter to supply
caller/source context -- single-shot on the ranker's region is insufficient; (2) the pool's
banks label is mislocated (head_lines 30/104/138-141, function _load; real sink is line 44),
so even a correct finding wouldn't score as a catch and the sink pre-exists in base -- a
change-scoped hook correctly stays quiet. Part of the pilot engine 0/5 is SZZ mislocation, not
a detection failure. GLM/abliteration fallback errors HTTP422 on the thinking param (client
fix needed; DeepSeek works).
Next: validate the 5 pilot labels (change-located & introduced?) before detection runs; then
size the agentic hunter on the sound targets; feed confirmed source shapes back as guarded facts.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified root cause: the pilot repos are libraries whose untrusted input is public-API
parameters, not HTTP requests. Python taint sources (ruleset/semantic/python.toml) are web/CLI
only; the parameter-as-source mode (_parameters_are_input) fires only for recognised entry
points, and the pilot vuln regions are entry:false -- so their params are deliberately not
sources. Widening that statically would reintroduce the WordPress/PMPro false-SQL-injection
flood the docstring documents. So the static engine cannot safely source library APIs on its
own -- which is exactly the seam for the maintainer's '2 guided by 1' directive: the LLM
adjudicates which library parameter is a trust boundary, the taint engine traces that
source->sink as proof (not LLM say-so), and repeated source shapes graduate to guarded static
facts. First increment (pending budget): LLM-guided-source mode on the 5 pilot regions,
measuring source naming, traced finding, catch lift vs regex 1/5, and false-alarm on ordinary
changes; staged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Inspecting head.findings (not ranker evidence) overturns the previous commit's read:
the dataflow/taint tier produced ZERO findings in the vuln families (path/injection/deser)
on all 5 repos -- only config_secrets (banks 1, socketio 2; launch-editor/graphiti/bedrock 0).
The tier1/score2.0 'sink' I cited was the RANKER's static candidate, not a confirmed taint
finding. taint_evidence_not_applicable = no source->sink flow was traced (banks: sink found,
no source). So ambiguous_line_correspondence is downstream and moot -- there are no
vuln-family findings to anchor, and the region-level witness fix (Option A) is refuted before
implementation. The LLM adjudication tier was disabled (model_calls=0).
Reprioritised: #1 LLM-tier pilot L1/L2 (now primary, staged, costs money), #2 taint
source/sink coverage + region ranking, #3 change-anchoring (only after tiers yield findings).
Instrument rule: caught the ranker-vs-finding misread before building the wrong fix.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Five public pool repos, vuln change each, clean host, 300s. The shipped engine added NO
catches over the regex floor. 3/5 repos finished their region queue (deferred=0) and still
emitted no finding, so change-scoping is not the path to catch. The engine ranked the vuln
region in all 5 and found the right sink on 2 (score 2.0), but withholds any finding it
cannot anchor 1:1 to changed lines -- all 5 hit ambiguous_line_correspondence ->
witness_unresolved -> coverage incomplete, suppressing even the scored sinks. Confirmed not a
shallow-clone artifact (correspondence is computed from both blobs).
Reprioritised engine levers: #1 change-anchoring robustness (region-level witness fallback),
#2 family ranking/taint coverage, #3 the LLM adjudication tier (the pilot is the case for it).
Release read: the pre-push net is NOT releasable on detection yet.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On a clean host the shipped engine runs dataflow (query 0->35s, selected 0->356 regions).
The evidence/taint batch (~225s) is invariant to warm session and graph preload; importCpg
is cheap (~35s). The real barrier is taint.sc's ~160s per-batch flow work, so build+evidence
~255s of a 300s budget defers 1009/1365 regions. Lever = change-scope the family/file set the
taint batch walks, not session/preload. Corrects the prior ~200s graph-load wall diagnosis.
Next decisive experiment: engine-on unseen pilot vs record-07 regex floor (catch 3/92).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Verified on the records: the push replay arms hardcode model="disabled",
max_model_calls=0, cost_usd=0 (NO LLM) and engine_covered=0 (engine never ran,
~200s wall). So the measured catch~2.5% is the quick pattern-rule tier ALONE --
grep-with-structure. Pattern rules alone will never be enough.
Reprioritize G2 after record 07 to an efficiency barrier analysis over the two
tiers that carry detection and both sat out:
- engine (Joern dataflow): E1 split the ~150s evidence-fixed (JVM/graph-load/
census); E2 does a warm session + graph preload at build kill it (decisive);
E3 analysis per-pair; E4 build vs repo size.
- LLM adjudication: L1 model latency + calls per push; L2 recall lift over
regex+dataflow on the pool.
Deliverable: a barrier table (budget x tier combo -> fits? measured recall/FP)
for a realistic pre-push window (<=120s target, 300s hard) -> minimal releasable
config. Change-scoping branch is a late add-on (trims only the per-pair tail),
not the lever. Also fixed a stray </content> tag in the brief.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Records the full G2 engine-detection finding on main (code attempt parked on
branch engine-change-scoped-evidence-pass, unmerged). The engine arm has ~200s
intrinsic fixed overhead -- ~48s CPG build + ~150s first-query graph-load+census
(measured uncontended: build 48s, evidence 243.79s over 1015 pairs, query 0) --
so a 300s budget cannot reach the flow queries. Pair-count change-scoping trims
only the ~86s per-pair tail; the warm-session env times out at startup contended
and spares only subsequent batches. Real fix is clean-host engine work (reliable
session startup + graph preload, or census-cost reduction). Parked to finish G0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Saved-artifact run: total=291s build=52s query=0s arbitrate=238.75s, and
query_seconds_by_kind[evidence]=238.54s. The evidence-tier pass is the whole
middle; affected_context + _scope_work are 0.21s. It tiered 1015 pairs at
~235ms/pair (contended) and exhausted the budget -> selected=0, deferred=1365
all deadline_exhausted. Secondary anomaly: ~50-100x the design's assumed
2.5ms/pair -- the evidence query is pathologically slow on a cold large CPG.
Increment 2's single target is the evidence pass (scan.py:338-359). Fork
recorded: (i) scope+bound the pass change-first, (ii) two-phase neighbourhood
pass, (iii) per-pair speedup. Must preserve reach-into-unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A first scoping attempt (restrict analyzed regions to changed files) was reverted:
it breaks a tested design invariant. The engine deliberately selects an unchanged
region's question when a changed region reaches it
(test_changed_dependency_reaches_unchanged_ranked_question;
test_change_does_not_replace_ranker_selection). _changed_files feeds _evidence_pass
only as contextFilter/contextPaths (itemise vs summarise), not to shrink the tier
population. Any scope must keep changed regions PLUS the regions they reach.
Also: increment 1 over-claimed. Increment 0 measured build=67s/query=0s but NOT
the split of the ~236s middle (evidence-tier pass vs affected_context/context
resolution vs _scope_work enumeration). Increment 2 must capture per_kind /
other_seconds / affected_context timing and cut the dominant phase while
preserving reach-into-unchanged -- instrument before fixing.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 4095 candidates are not an in-scan artifact: push/runner.py:345 discovers
head_regions across the whole head tree, then runner.py:397 passes them wholesale
to scan_repository with change_context threaded in but unused for scoping. The
scan enumerates all regions x families (scan.py:167/364) and, with
order_by_evidence=True, evidence-tiers every pair before analysis -> the ~236s
from increment 0 -> deadline at query_seconds=0.
Fix locus (increment 2): scope the analyzed region set to context.changed_paths +
dataflow neighbourhood (regions.py:279 primitive exists; compare_targeted_base is
precedent). Keep the CPG whole-repo so taint can trace into the change from
unchanged code; only the question set is change-scoped.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Timed live bedrock replay: build_seconds=66.69, query_seconds=0.0, total 303s,
4095 candidate questions all analysis_not_started=deadline_exhausted, selected=[],
regions_scanned=0. ~67s builds the CPG; ~236s enumerates/ranks/prepares 4095
repo-wide candidates for a 3-file change, then the deadline hits before any query
runs. The CPG build is not the bottleneck; the over-broad candidate set is ->
change-scoping is the confirmed lever (and compounds with the correspondence fix,
since per-candidate prep includes correspondence work).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Track the engine-detection lever found while driving G0. Live diagnosis on the
frozen pool: the engine's "no finding" on unseen changes is usually
analysis-never-runs -- it ranks repo-wide candidates and defers them all
(deadline_exhausted) before analyzing the pushed change (bedrock injection,
303s, selected=[], ranked sites unrelated to the change). Lever: seed analysis
from the change's declarations + dataflow neighbourhood (rank.py/mapping.py) so
it runs within budget.
- .kiro/specs/engine-change-scoped-analysis/brief.md: problem (measured),
hypothesis, anchors, and checkable increments (0 confirm build-vs-analysis
cost split -> 1 locate selection/budget -> 2 seed from change -> 3 soundness
+ tests -> 4 live re-test -> 5 measure delta on the engine arm).
- roadmap.md G2: active workstreams (change-scoped analysis; the done-but-
necessary-not-sufficient line-correspondence fix), for progress tracking.
Honest scope: G2, gated on the pool metric; not part of the G0 quick-rule
record-07. Coverage also needs the capability registry (Task 8.3); arm-scale
measurement needs the kube lane.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The correspondence mapped an equal-region base<->head line pair only when the
line's content was GLOBALLY unique in the file, so common lines (`}`, blanks,
imports) tripped `ambiguous_line_correspondence` and the finding's anchor was
discarded -> the operation came back `unmapped` -> novelty `unknown` ->
`comparison_unknown` -> the engine finding was suppressed (a direct cause of
"near-blind to fixes").
Replace the global-uniqueness gate with run-aware positional anchoring: segment
the equal-region pairs into contiguous runs (strictly +1/+1, as _equal_line_pairs
emits), and map a pair when its content occupies exactly one old run and one new
run. Duplicates WITHIN a contiguous run are positionally forced and now map;
duplicates ACROSS runs split by an edit are aligned arbitrarily by git and stay
withheld + flagged (soundness preserved -- the moved-repeated-operation guard
still passes, so no false `new`/identity claims).
Tests: update test_change_context_unknown_base_binary_and_ambiguous_lines_are_
unresolved (its two in-run sinks now anchor); keep the moved-operation soundness
test; add intra-run-maps, cross-run-withheld, and distinctive-line-anchors cases.
163 push tests pass, ruff clean.
Honest scope (design.md amendment): this removes one real cause but is
necessary-not-sufficient -- live-testing a messy 14-file Java change showed
coverage stays `incomplete` due to other, dominant blockers (capability_unavailable
until Task 8.3; the engine produced no actionable defect; eligibility/projection
gaps). Engine coverage on unseen diffs needs those too, not just correspondence.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Fold the night's learnings into the roadmap, steering and the
unseen-repo-evaluation spec so the process is reproducible and the
project is "ready now":
- roadmap.md: a 2026-10-11 state table (pool p1 frozen; harvest hardened
+ bucket-first; replay-image provenance pinned; baseline-on-kube next)
and a 5-step readiness plan to record 07.
- storage-streaming.md: RustFS-over-local as the default (local disk is
the scarce resource); the implemented ousast-corpus layout; replay
outputs belong on RustFS; the AX store endpoint is operator-owned.
- design.md: an "Amendments (2026-10-11)" section (bucket-first
persistence, ~2.2% rate-bound yield, machine-label record 07, pinned
replay image provenance).
- tasks.md: a folded-in amendment block (pool p1 frozen + freeze recipe;
bucket-first; CI replay image; machine-label baseline; kube lane +
ax/store as operator-owned infra).
Facts stated are verified (measured yield, freeze digest, GHCR anon pull,
batch.py's endpoint check); infra capability/policy is left to the
operator, not assumed. The frozen pool artifacts themselves are not
committed here (a separate samples/licensing decision).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
