Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 5.3, 2.1, 2.2, 2.3, 7.2 of ai-service-plane.
- plane/tasks/verify.py: the batched hunt of verify_batched.py moved into
the package (system prompt, one hunt per file, up to 6 candidates, 6
steps, candidate-anchored sites). New against the reference: each
candidate's known callers from facts.json in the prompt (capped at 8),
tool turns and usage per hunt in units.jsonl, per-unit resume (rows
with "error" are re-asked), pass label from env, MeteredClient wrapper:
BudgetExhausted -> unfinished (exit 3), AccountError -> failed with the
provider message (exit 2); summary.json is written last, names the bound
Model and the summed usage, and is done only when every unit finished.
- plane/tasks/agree.py: verify_batched.agreement over two pass outputs,
agreed.json / disputed.json, per-candidate cost and turns, declared-site
match via benchmarks/independent/evaluate.matches_v2 loaded by file;
metrics block in summary.json; no model calls.
- tests/test_plane_verify.py: a scripted usage-recording hunter client;
callers line, resume, 402, budget stop and resume, model and summed
usage, unpriced usd None, both main() env contracts, agreement metrics.
- model/audit.py + module-audit manifest: SPEC_OWNED rows for plane.tasks,
plane.tasks.verify, plane.tasks.agree; manifest regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/tasks/repo_facts.py computes, from source text only and without a
model or git, the product files of a workspace (tests, vendored trees,
node_modules, dist/build, minified bundles and dot-directories excluded via
the layout facts plus the model_sinks.py filter), the functions each file
declares (Python, PHP, JavaScript/TypeScript, Java, Go, Ruby declaration
patterns extended from sink_candidates.py) and, per candidate function, the
call sites in other product files with path, 1-based line and enclosing
function, capped at 40 and sorted (Requirement 5.1). Output is byte-identical
across runs over the same tree (Requirement 5.2).
Under the task contract the unit is one file: a row goes to units.jsonl as
each file finishes and a restart reuses the rows computed for the same
candidate set (Requirement 2.2); facts.json is assembled from the rows and
summary.json is written last, refusing `done` unless every unit finished.
The entrypoint reads OUSAST_WORKSPACE_DIR, OUSAST_OUTPUT_DIR and optional
OUSAST_INPUT_CANDIDATES only (Requirement 2.1) and exits 0 on done, 2 on
failed with the traceback recorded in summary.json.
tests/test_plane_repo_facts.py: a three-file tree with a tests/ file, a
vendored file, a node_modules bundle and a dot-directory; cross-file callers
with line and enclosing function, same-file and declaration lines excluded,
the 40 cap, byte-identical second run, resume that recomputes only the
missing unit, a crash that leaves `failed` with the unit counts and resumes
to `done`, and the declaration patterns per language.
The two new modules are registered as spec-owned in the module audit (their
importer, the runner, lands in task 3) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kind cluster + registry, Agent Substrate, ax control plane via ko, one files-only
smoke Task; helpers run from their own checkout because they resolve paths from
the git toplevel. Footprint and README follow once the cluster is up.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add openultrasast.plane.manifests: frozen dataclasses and loaders for ax's
Task, Workspace and Model kinds (ax.io/v1alpha1, ax's fields only) and the
Run kind (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs,
budget, serialize). Every violation raises ManifestError naming the kind,
metadata.name and field path: unknown fields at any depth, wrong apiVersion
or kind, a provider outside google/anthropic unless deepseek is declared via
the openultrasast.io/provider-extension annotation, a Run with a dependency
cycle, an input naming no task, and a Run task naming no loaded Task.
load_manifests reads multi-document YAML files and indexes by kind and name.
PyYAML becomes the one declared core dependency; the two zero-dependency
guards now pin exactly that, and the module audit registers the plane
package under the ai-service-plane spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decisions 2026-09-29: ax (kind + Agent Substrate + ax controller on
this host) is the only executor; the Run layer carries dependencies, artifacts,
budgets and attribution until ax has them. Requirement 7 declares routing per
Task (one Model per Task, no runtime switching) and a token attribution table
from provider usage fields. Task 0 (bring-up, doctor), runner artifact delivery,
ax-backed reconciler with a fake-ax test seam. harnessx-removal brief records
the HarnessX inventory; removal is gated on the increment's measurement.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
MeteredClient wraps a ChatClient, sums the provider's usage rows per call,
prices them from a Model manifest's parameters (cache-hit, input, output per
million), and raises BudgetExhausted before a call once the usd or calls
ceiling is reached. A 401/402 or "Insufficient Balance" failure becomes
AccountError with the provider's message. An unpriced model reports usd None,
never zero, and refuses a usd budget; summary() gives the spend block of a
task's summary.json.
The two plane modules are registered as spec-owned in the module audit (their
consumers land in tasks 3-4) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine tasks in dependency order: manifests, budget metering, runner contract,
reconciler and CLI, repo-facts, verify and agree, the increment's manifests, the
measurement against the recorded batched check, handover.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Package src/openultrasast/plane/: manifests (ax kinds + Run), a reconciler under
500 lines with persisted status and resume, an ax-contract runner, budget metering,
and task entrypoints. First increment: repo-facts (source-only callers, no model),
verify (the batched hunt with callers in the prompt, per-unit resume), agree.
Measurement on the 46-candidate set against the recorded batched check.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Six requirements from the brief: ax kinds plus a Run kind; isolated, idempotent,
resumable tasks with metered budgets; a local reconciler under 500 lines with no
pipeline logic; ax's runner contract; repo-facts as the memory layer; and a
measured first increment (repo-facts + verify on the 46-candidate set: at least 16
of 20 declared sites, under $0.022 per candidate for two passes).
Phase: requirements, not yet approved.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the main session: 90% of development tokens were tool I/O. Three
tools from the Portal pattern:
- read_guard.py: PreToolUse hook that denies a whole-file Read over 350 lines and
points to a ranged read or the bulk reader (pipe-tested: deny, range, small).
- bulk_read.py: answers a question about files with the cheap chat model, citing
path:line, under a spend ceiling.
- token_report.py: where a session's tokens go, from its transcript.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Adopt the design of google/ax (Task / Workspace / Model manifests, a reconciler,
isolated resumable execution, a runner contract), not yet its software: it needs
Kubernetes and Agent Substrate, lists only Google and Anthropic providers, and has
no pipelines, artifacts or budgets. We add a thin Run layer for those.
Each pipeline concern (classify, triage, repo-facts, verify, judge, recheck, score)
becomes a declared task with its own inputs, outputs and budget. That bounds the
tokens per task and lets each be developed alone. Decisions listed for the
maintainer.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Exploratory, 7 of 17 cases, stopped when the account emptied:
- source-only scope: a model reading every product file puts the vulnerable
function among the candidates in 15/17 (ranker 1/17, vocabulary 2/17);
- per-file triage + per-file 6-step hunts + two-pass agreement: 12/15 candidate
cases on the validation set; 4/7 on the full run, two more in one pass of two;
- precision: 5 pgAdmin agreed findings read against source, 3 true (privileged),
2 false with self-contradicting rationales; agreement cannot catch an error
both passes make;
- cost: $0.009-0.022 per candidate for two passes.
Four instrument defects fixed on the way (model id, provider routing, JSON-mode
final turn, unbounded tool output) void the earlier hunter number. No gate is
met; qualification needs population v3 and a protocol that states the judge
stage, the pass rule and a config-family question.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- verify_batched.py --fixed asks the agreed candidates again at the fixed pin
(no triage); a function the fix removed is recorded as removed.
- score_batched.py applies protocol v2's matching to the batched output
(vulnerable pin, and the fixed pin where rechecked) and draws the precision
sample.
- A finding's site is anchored to the candidate operation asked about; the
model's own reported location is kept in `reported_at`. ReactPress's confirmed
findAll was reported at the controller file and failed to match its declared
site. The scorer anchors older records the same way.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".
On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
winml is triaged out as "constant": a CORS literal, which the reachability
question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
with several candidates, 2.3x cheaper per pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.
Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.
The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--declared-sites-only, --candidates-from <jsonl> and --results-suffix, so the
efficiency change can be validated on the 180 already-verified pgAdmin candidates
and the 15 declared sites without touching the main results.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs
$0.0139 per candidate, 90% of the spend, because the conversation is re-sent on
every turn and grows with each tool result.
- Context files now precede the question in the hunter's first message. DeepSeek
bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first,
every candidate asked about the same file (about 5 per file) broke the prefix on
its first line and paid for the file again. A test pins the order.
- The verifier runs each candidate at 3 steps and repeats it once at 6 only when
the hunt spent every step on tool calls without a verdict. On the seven declared
v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3.
- Per-candidate token usage (prompt, cached, completion, calls) and escalation
are recorded, and each case reports its cache share, so the next efficiency
decision is made on numbers.
Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1)
waits for account credit: DeepSeek returns 402 at the moment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The rerun hit DeepSeek HTTP 402 ("Insufficient Balance") after 180 pgAdmin
candidates. Every later call failed at $0, and the runner wrote sixteen cases as
complete with nothing asked, because an unverified candidate counted toward
completion. That is a clean-looking zero from an instrument that could not call
the model.
Now:
- 401/402 or "Insufficient Balance" stops the run at once, exit 2, nothing
recorded;
- only verified candidates count as done, so unverified ones are asked again on
resume;
- a case is complete only when every candidate was verified.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
FUXA failed every hunt with HTTP 400, and neither run could say why. The client
discarded the response body. With the body shown, the reason was "This model's
maximum context length...": FUXA ships minified bundles whose single lines run to
megabytes, and grep_repo returned a matched line whole.
- grep_repo and find_refs quote a matched line (240 characters), never the whole
line.
- Every tool result is capped at 16 x max_chars, with a truncation marker.
- OpenRouterChatClient errors carry the provider's response body (first 500
characters). That would also have exposed the rejected model id in one call
instead of eighteen days.
After the fix, FUXA candidates verify normally at $0.004-0.013 each. A test covers
the megabyte-line case.
verify_sinks.py: skip futures cancelled at the spend ceiling instead of reading
them (the run crashed with CancelledError).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.
Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.
The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.
Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Each grep_repo and find_refs call re-enumerated the tree (walk, ignore rules,
vendored checks) and re-read every file. The v2 sink verifier spent almost three
hours on its first case (pgAdmin, about 1,800 files) at 108% CPU with one open
connection. It was CPU-bound in these tools, not waiting on the model.
Source lines are now memoised per root, keyed by path and modification time. A hit
is re-validated file by file, because an edit inside a subdirectory does not touch
the root's mtime; a test covers one enumeration and a detected edit.
verify_sinks.py now runs candidates in worker processes, not threads, and appends
each candidate's result to a per-case log, so a stopped run resumes where it left
off. GoPay: 12 candidates in 14 s, the real site found.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Per candidate function, one bounded tool-hunter pass (custom prompt, the file as
context, 3 steps) asks whether attacker-controlled data reaches the operation
without a sufficient guard. It reports only with a stated source-to-sink path.
Hard spend ceiling, parallel workers; one run per pin; the fixed pin is rechecked
only for detected sites. Exploratory on v2.
Pilot: on GoPay, the one real site was reported in 3 of 4 runs and nothing else
out of 12 candidates.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Source-only selection on the v2 vulnerable pins, compared on the same cases:
- ranker top-20 entry points: the vulnerable file was in scope in 1/17;
- shipped-vocabulary sink-first: the declared site was a candidate in 2/17;
- model-classified sinks (deepseek-flash, no vocabulary): the declared site was a
candidate in 15/17, and a fix or site file in 16/17. Every chunk was classified;
$14.42.
Misses: User Registration (file in scope, not the function) and Sulu (JCR-SQL2
query builder). The cost of this reach is volume: 3-1037 candidate functions per
repository, 4,762 in total. The next stage must decide which candidates are fed by
attacker data without a guard. v2 is spent for tuning; this qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This asks DeepSeek, per product source file (in chunks of 400 numbered lines),
which functions perform the family's dangerous operation on non-constant data. It
is scored like sink_candidates.py: is the declared vulnerable function among the
candidates? A failed chunk counts as unclassified, never as "no sinks". The spend
ceiling is enforced across worker threads.
Pilot: GoPay found its declared site for $0.03. Estimated full v2 run: ~$14-18.
The maintainer raised the ceiling to $20.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is a source-only diagnostic: no engine and no model. At each v2 vulnerable
pin, every call matching the family's shipped sink vocabulary becomes a candidate,
attributed to its enclosing function.
- The declared vulnerable function is a candidate in 2/17 cases (Medplum, GoPay);
the ranker's top 20 managed 1/17.
- A fix-changed or site file is among the candidates in 8/17.
- Candidates per repository range from 2 to 904 functions.
The misses are vocabulary, not direction. Examples: wp_redirect, download_url,
spawn(shell:true), ky(), requests.Session().get, sympy parse_expr, Jinja
from_string().render, Mongo collection.find, JCR-SQL2 createQuery, TypeORM
andWhere template strings, and CORSMiddleware(allow_origins=["*"]) are not named.
v2 is spent for tuning, so this informs design and qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The run follows hunter-protocol-v2.md: deepseek-flash, the ranker's top 20
regions per pin, protocol v2 scoring, $8.88 spent.
- Recall: 1/17 (Wilson 0.01-0.27).
- Precision: 2/14 sampled (both true positives need a privileged role, one of
them GoPay's CVE).
- Fixed-side alerts: 1. Benign alerts: 16.
- Stability: 73% of findings differ between two identical runs.
The deciding observation: the ranker's top 20 contained the vulnerable file in 1
of 17 cases, and in that case (GoPay) the hunter detected it. With 4 tool steps
per hunt it cannot search its way to code it is not handed. The result therefore
says less about the model than about scope selection, which sits upstream of the
engine too.
Instrument failures, reported as failures: every FUXA pin and three Directorist
pins (DeepSeek HTTP 400 on every hunt; neither offered a region of its family).
Scorer fix: protocol v2 defines benign alerts and the precision pool over the
CASE FAMILY, but score_v2 pooled every family. The engine's dump emits only the
case family, so its recorded result (21adc40) is unchanged by this; rescoring
reproduces 3 benign alerts and 49 pooled findings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the deciding experiment of the strategy review, fixed before any v2 pin is
hunted.
- Hunter: the unchanged tool hunter (default prompt, tools and 4 steps) on
deepseek-flash.
- Scope: the ranker's top 20 regions for the case family, in four hunts. The model
never sees the fix diff or the declared sites.
- Families: a fixed keyword table maps the model's family to the protocol's three.
- Scoring: exactly protocol v2's scorer.
- Cost: a hard $10 ceiling.
hunt.py runs it and scores it (--score). A pilot on a v1 development case
(mongo-express) found the vulnerability at lib/bson.js:60, at $0.031 per pin,
alongside a lot of speculative noise.
Transport fix found by the pilot: the DeepSeek endpoint the hunter builds was
unpriced, so the cost meter read $0.00 for any number of calls. It is now priced.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every model call has failed since 2026-09-10 (83ab533):
- The default detector and judge model was `deepseek-v4.1-flash`, the product name.
The API rejects it with HTTP 400 ("The supported API model names are
deepseek-flash, deepseek-v4-pro"). The judge and pipeline record an endpoint
failure as a degradation, never a verdict, so nothing failed loudly.
- `tool_hunter.resolve_hunter_client` resolved OpenRouter only. In this project
DeepSeek is the chat provider and the OpenRouter key is scoped to embeddings, so
the hunter could not reach a chat model at all; the call came back 401.
The defaults are now `deepseek-flash` (priced). The hunter resolves DeepSeek first
when its key is set, and OpenRouter stays as the explicit or fallback route.
Verified live: plain completion and a tool call through both the endpoint resolver
and the hunter client.
Not affected: the independent evaluations (finding_dump runs with
max_model_calls=0) and the 2026-09-06 hunter measurement, which predates the id.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 (1/11) and v2 (0/17 recall, 0/17 sampled precision, 12% completion) show that
fixes learned on one population do not transfer to the next. The causes are
breadth: framework-shaped entry points, module boundaries the frontends do not
resolve, name-matched sinks colliding, and scale.
Options:
- A: continue engine-first;
- B: the model proposes, the engine proves;
- C: narrow the product to one ecosystem.
Recommendation: first run the unchanged tool hunter on the v2 pins as the deciding
experiment.
For maintainer decision; no approved spec changes.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was frozen at a75b9f6. 17 cases, 102 scans (vulnerable and fixed pins
twice each), scored exactly as pre-registered.
- Recall: 0/17 (Wilson 95% 0-0.18). No unstable detections.
- Precision: 0/17 sampled (0-0.18). Every sampled finding is false: regex
match/exec read as sinks, int casts, ORM statements, client-SDK code, the
process's own argv, a generated database name, a registered-URI check.
- Completion: 13206/108374 questions (12%).
- Fixed-side alerts: 0, trivially, since nothing was detected.
- Benign alerts: 3.
- Stability: 2 of 100 findings differ between runs.
The zero is not an instrument artefact: FUXA, GoPay, cct-studio, ReactPress, User
Registration and Directorist completed questions in the sink file itself and still
found nothing.
Instrument failures, reported as failures: LMDeploy (every taint batch failed on all
six pins) and Froxlor's two benign scans.
This record is committed before any analyzer change informed by v2.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any v2 case is scanned. It has two changes from v1, both chosen by
the maintainer from lessons of the v1 run:
- declared sites: a finding at a `path::function` the advisory names is credited,
as well as one on the fix's changed lines;
- two runs of the vulnerable and fixed pins: detection needs the finding in both
runs, a fixed-side alert in either counts, and the difference between runs is
reported as stability.
evaluate.py takes --population. v2 scans six pins per case and scores with
matches_v2 / score_v2. v1's scoring is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was not run. freeze.py read each case's source at both pins:
- both pins are commits in the repository and in the right order;
- the declared sink site's file is present at the vulnerable pin, and its declared
functions appear in it;
- the fix touches a declared site file (Budibase's sites reordered so the changed
file, queries.ts, comes first).
Benign controls:
- 12 are reviewed automatic proposals.
- 3 automatic proposals were rejected as not benign: qwed's merged a ReDoS fix,
ReactPress's corrected a SQL column whitelist, and Sulu's was a branch merge that
can carry the fix.
- Those three, plus cct-studio and Directorist (no automatic proposal), were chosen
by hand. Directorist's comes from before the vulnerable pin, because only 3
commits follow its fix.
freeze.py now accepts a recorded benign control instead of reporting that none was
proposed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 was evaluated and then used to retune the analyzer. A retune can only be
qualified on a population that did not inform it.
The candidates came from web research alone (research-2026-09-26.md):
- advisories were taken newest-first per language under fixed criteria, never by
how easy they look;
- 261 advisories were examined and 224 rejected, each with a reason;
- every SHA pair was verified through the GitHub API;
- no clone and no scan.
The maintainer selected the rule-based "balanced" set: 5 Python, 6 JS/TS, and 6 PHP
of which 4 are WordPress plugins; 10 injection, 6 destination and 1 config.
Each case declares the advisory's sink sites before any scan, for protocol v2.
freeze.py takes --population and writes freeze-vN.json. The reservation tests now
cover every population-v*.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This reverts commit cc885f0. It failed the acceptance bar set before the run (each
recall pin within +2 findings, completion unchanged):
- WP Statistics: 16 -> 31.
- PMPro: 16 -> 26, and completion 378/382 -> 103/382. Six prior findings were lost
to the budget; CVE line 936 survived.
- YesWiki: completion 2798 -> 1968, still not detected.
Two causes:
- Cost: wrapper confirmation runs a flow query per parameter of every sink-bearing
method, repeated at each engine start of a portioned scan.
- Precision: a helper whose parameter reaches a sink under its own escaping (for
example `getMemberOrderByPayPalToken(sanitize_text_field(...))`) is not a sink
for its caller.
Doing this properly needs summaries that carry the wrapper's internal
sanitisation, computed once per scan, not per request batch.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the second attempt at the YesWiki shape: SQL run through
`$this->dbService->query($query)`, whose body is the `mysqli_query` the facts name,
through a receiver the frontend cannot type. The first attempt was reverted in
c7e3a05.
A method is now a wrapper for a parameter only when the engine's own dataflow
inside the method carries that parameter to a fact sink's argument, on a plausible
path that no cleansing or guard discharges. The guard rails:
- one level only;
- nothing under a bundled library tree (lib/, vendor/, node_modules/, ...);
- a name counts only if every project method with that name is a wrapper;
- wrappers are asked in the flow pass only, never in the evidence pass, so no
region's selection or pruning changes;
- at a wrapper call, only the confirmed argument is asked.
Three Joern-gated tests cover the direct shape (reported), a literal, an
intval-casting helper and a level-2 feeder (not reported), a harmless same-named
method, and a wrapper inside lib/. Each fails when its rule is removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This reverts commit 54c9dfc. The recall-pin check (p16) showed a large precision
and completion regression, although every recall target survived:
- MW WP Form: 3 -> 60 findings. Form-field renderers became sinks.
- WP Statistics: 16 -> 40.
- PMPro: 16 -> 65, 20 of them inside bundled Braintree/Stripe code. Completion fell
to 390 of 1189 questions, because wrappers made many more regions look
sink-bearing.
- YesWiki: completion fell from 2798/2807 to 216/2807.
The structural summary was too loose. Any identifier in a sink argument counted as
fed by a parameter if any assignment anywhere in the method mentioned it, with no
order or dominance. It also ignored vendored library code and cascaded three
levels by name. A redesign needs dataflow-confirmed, level-limited summaries that
leave the question count alone.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Real code rarely calls the driver where it builds the query. YesWiki builds its
INSERT in `FormManager::create($data)` and runs it through
`$this->dbService->query($query)`, whose body is the `mysqli_query` the facts name.
The only sink the facts knew sat in a helper that no request reached directly.
A method whose parameter feeds a family sink is now a sink for that argument,
reported at the call site. The summary is structural, to stay cheap and
deterministic. It is not a flow query per parameter, which would repeat at every
engine start of a portioned scan. The parameter must reach the sink call's
argument directly or through at most two local assignments, with at least one
occurrence outside every cleansing call. It iterates up to three levels, so a
wrapper of a wrapper counts.
Calls resolve by name, because an injected receiver has no type the frontend can
see. A name therefore counts only if every project method with that name is a
wrapper. At a wrapper call, only the summarised arguments are asked.
Two Joern-gated tests cover:
- a request value into a second-level wrapper, reported at the controller's call;
- direct use, a literal, and an `intval`-casting helper, which are not wrappers;
- a harmless same-named method, which disqualifies the name.
Both fail with their part removed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Reading /proc/<pid>/stat for a process that is reaped during the read raises
ProcessLookupError (ESRCH), not FileNotFoundError. The previous race fix caught only
the latter, so the hanging-process-group test failed twice today under full-suite
load. It passed alone. The disappearance is exactly what the test asserts.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
p14 showed PMPro's functions.php:906 discharged by `elseif (in_array('E', $levels))`.
That checks whether a constant is in `$levels`; it says nothing about `$levels`. The
guard counted a carried variable in any argument. Only argument 1, the tested value
(`in_array`'s needle, the URL given to `url_has_allowed_host_and_scheme`), is
validated now. The allowlist test gains the haystack shape as a reported control.
The test fails when any argument counts.
The other p14 removals were right: WP Statistics export_data x2
(`if (!in_array($table, $allow_tbl)) $table = false;`) and PMPro memberorder.php:1633.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Ultimate Member's fixed pin kept one alert. The fix writes
`$this->sql_order = " ORDER BY u.{$sortby} "` inside
`elseif (in_array($sortby, $allowed, true))`, and after the if/elseif chain it
passes the field through `apply_filters(...)` into a second assignment. The guard
was judged only at the flow's final node, which is outside the branch.
A guard is now also judged at every point where the path stores the value (an
assignment target). A value stored while the check held carries the check forward,
because the rest of the path derives from that store. A mere read inside a branch
is not a candidate: the engine lists uses in branches the flow does not depend on.
The Joern-gated test gains the shape and its control: the same store in the
unguarded else branch, through the same filter, is still reported. The test fails
when only the end is judged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Three of the four fixed-side alerts left in the independent population had one
cause: the fix was a check, and checks were not modelled.
- wger's fix is `url_has_allowed_host_and_scheme`: `if not ...(next_url):
next_url = reverse(..)` and `if next_url and ...(next_url): return redirect`.
- Ultimate Member's fix is an allowlist: `in_array(strtoupper($order), [...],
true) ? $order : 'ASC'` and `elseif (in_array($sortby, $allowed, true))`.
(The fourth, Signal K, is `assertAllowedHost`, a project-local function. It is not
covered.)
Sanitizer facts can now say `guard = true`. A guard is a check, never a cleansing
call on a path. A flow is discharged only when a guard tests a variable the path
carries, and the flow is in one of these places:
- inside the branch whose condition requires the check. The guard may be combined
with `&&` only;
- past a failure test, a single `!` combined with `||` only, whose branch exits
(return, exit, die, wp_die, throw) or overwrites the value, and whose condition
dominates the flow's end;
- through a ternary whose condition requires the check, where the other arm cannot
carry the value. The engine can enter through the condition's own copy of the
value, which never becomes the result.
Facts: `in_array` (PHP) and Django's `url_has_allowed_host_and_scheme` /
`is_safe_url` (Python). A row now consults guards even when a request lists no
sanitizers.
Two Joern-gated tests cover the discharged shapes and, as controls, an unguarded
redirect, a guard on a different variable, a failure that is only logged, the
unguarded else branch, and the ternary arm the check does not select. Both tests
fail without the guard.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
