agree's declared_sites_matched counts candidate coverage, not detection. Recounted
from agreed.json rows, plane passes flagged 17 of 20 declared sites against the
reference's 19 (budibase read and fuxa getDaqValue missed by both passes). The
earlier "detection unchanged" reading was wrong.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
validation-46 ran done on google/ax: 60 tasks, 357 DeepSeek calls, $0.83 with
recorded triage ($0.0194 per candidate, reference $0.022: cost met). Declared
sites agreed 14/20 against the reference's 16/20: recall criterion not met.
Both runs find 19/20 sites in at least one pass; the two-pass agreement
flicker moved three sites. Design revisit before task 9.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's egress gateway denies an actor without an EgressPolicy, so
no task could deliver its artifacts or reach a model. egress.py grants each
task exactly: http/80 to ousast-receiver.ax-system.svc.cluster.local, and
TLS passthrough/443 to the bound Workspaces' https Git hosts and the bound
Model's openultrasast.io/egress-hosts (api.deepseek.com for deepseek-flash).
No IPs, no wildcards, no https (MITM) rules.
The reconciler writes the policy with kubectl-ate between ax apply and
resume, reads it back, waits OUSAST_EGRESS_SETTLE (11 s, the gateway's
cached deny), and after ax delete task confirms the policy went with its
actor (kubectl-ate has no delete verb; the store cascades).
The receiver is a selector-less Service (ops/ax/receiver-service.yaml.tmpl,
applied by up.sh together with the no-build agentgateway egress gateway)
on port 18090 of the host. Measured live: the gateway checks the policy on
Host but connects to the address the actor dialled, so the runner dials the
Service's ClusterIP (OUSAST_ARTIFACT_DIAL) with the receiver's name as Host.
The runner now labels its delivery with the Run's task name (OUSAST_TASK),
which is what the receiver accepts; the ax Task name was refused with 403.
The Receiver moved to egress.py to keep reconciler.py at 460 lines.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Agent Substrate's golden-snapshot boot runs the same image with the same task env,
so the runner no longer starts its command by itself: after the workspaces are
ready it waits for POST /ousast/v1/start (run/task checked, once per boot, refused
after a delivered run). The reconciler sends it through svc/atenet-router with
ate-target-actor, which never addresses the golden actor (plane/router.py), and the
bound Model's secretKey variable travels only in that body, into the child's env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Model deepseek-flash (declared provider extension, endpoint prices in
parameters, secretKey naming the DEEPSEEK_API_KEY secret), digest-pinned
Task templates repo-facts / verify / agree, and a generator
(plane/generate.py, `ousast plane workspaces --validation-set ...`) that
emits per case the vulnerable-pin Workspace, a files-only inputs Workspace
(recorded candidates after the recorded triage, declared sites, triage),
per-case Tasks and one Run validation-46: facts -> verify a, verify b ->
agree for the 15 cases, 46 candidates (43 kept), budgets 1.5x the recorded
per-pass share. A Run entry cannot add a binding or env, and Run.inputs
names only task artifacts, so the case is fixed by per-case Tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ai-service-plane task 3, Requirements 4.1-4.4 (with 2.2 and 2.3 honoured by the
exit-code contract):
- src/openultrasast/plane/runner.py: the ax-task-runner entrypoint. Reads the Task
from AX_TASK_YAML and the bound Workspaces from AX_WORKSPACES_YAML (task-1 loaders),
serves /healthz and /readyz on port 80 (AX_RUNNER_PORT; 0 picks a free port,
AX_RUNNER_HTTP=0 disables it for unit tests), materialises each workspace at its
bound path (spec.files verbatim; spec.git at <path>/<name>, exported from the
OUSAST_CASE_CACHE checkout whose origin matches, else a shallow clone at the pin;
a repo already present on a resumed volume is kept) and only then answers 200 on
/readyz. Runs spec.command by importing openultrasast.plane.tasks.<name> with the
remaining elements as argv; sets OUSAST_WORKSPACE_DIR from the first binding when
the Task's env leaves it unset. A crash writes summary.json with status failed and
the traceback. When OUSAST_ARTIFACT_URL is set, the whole OUSAST_OUTPUT_DIR is
POSTed as an uncompressed tar (Content-Type application/x-tar, X-Ousast-Task,
X-Ousast-Run) with three attempts and backoff; a failed delivery rewrites
summary.json to failed and exits 2. Exit codes: done 0, failed 2, unfinished 3.
- plane/Dockerfile.runner: python:3.12-slim + git, `pip install .` (no extras), a
shell wrapper at /usr/local/bin/ax-task-runner exec'ing the module; built and
loaded into kind by ops/ax/up.sh, not by the tests.
- tests/test_plane_runner.py (+ tests/plane_stub_task.py registered as the task
module): env contract on a files workspace, readyz 503 then 200, delivery to a
local receiver with the headers and tar checked, delivery failure -> failed and
exit 2 after three attempts, crash -> failed with traceback, exit codes per
status, case-cache export at the pin, shallow-clone fallback, resume keeps the
checkout.
- model/audit.py: SPEC_OWNED row for plane.runner; module-audit measurement
regenerated (122 modules, no orphans, note kept).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 5.3, 2.1, 2.2, 2.3, 7.2 of ai-service-plane.
- plane/tasks/verify.py: the batched hunt of verify_batched.py moved into
the package (system prompt, one hunt per file, up to 6 candidates, 6
steps, candidate-anchored sites). New against the reference: each
candidate's known callers from facts.json in the prompt (capped at 8),
tool turns and usage per hunt in units.jsonl, per-unit resume (rows
with "error" are re-asked), pass label from env, MeteredClient wrapper:
BudgetExhausted -> unfinished (exit 3), AccountError -> failed with the
provider message (exit 2); summary.json is written last, names the bound
Model and the summed usage, and is done only when every unit finished.
- plane/tasks/agree.py: verify_batched.agreement over two pass outputs,
agreed.json / disputed.json, per-candidate cost and turns, declared-site
match via benchmarks/independent/evaluate.matches_v2 loaded by file;
metrics block in summary.json; no model calls.
- tests/test_plane_verify.py: a scripted usage-recording hunter client;
callers line, resume, 402, budget stop and resume, model and summed
usage, unpriced usd None, both main() env contracts, agreement metrics.
- model/audit.py + module-audit manifest: SPEC_OWNED rows for plane.tasks,
plane.tasks.verify, plane.tasks.agree; manifest regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/tasks/repo_facts.py computes, from source text only and without a
model or git, the product files of a workspace (tests, vendored trees,
node_modules, dist/build, minified bundles and dot-directories excluded via
the layout facts plus the model_sinks.py filter), the functions each file
declares (Python, PHP, JavaScript/TypeScript, Java, Go, Ruby declaration
patterns extended from sink_candidates.py) and, per candidate function, the
call sites in other product files with path, 1-based line and enclosing
function, capped at 40 and sorted (Requirement 5.1). Output is byte-identical
across runs over the same tree (Requirement 5.2).
Under the task contract the unit is one file: a row goes to units.jsonl as
each file finishes and a restart reuses the rows computed for the same
candidate set (Requirement 2.2); facts.json is assembled from the rows and
summary.json is written last, refusing `done` unless every unit finished.
The entrypoint reads OUSAST_WORKSPACE_DIR, OUSAST_OUTPUT_DIR and optional
OUSAST_INPUT_CANDIDATES only (Requirement 2.1) and exits 0 on done, 2 on
failed with the traceback recorded in summary.json.
tests/test_plane_repo_facts.py: a three-file tree with a tests/ file, a
vendored file, a node_modules bundle and a dot-directory; cross-file callers
with line and enclosing function, same-file and declaration lines excluded,
the 40 cap, byte-identical second run, resume that recomputes only the
missing unit, a crash that leaves `failed` with the unit counts and resumes
to `done`, and the declaration patterns per language.
The two new modules are registered as spec-owned in the module audit (their
importer, the runner, lands in task 3) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add openultrasast.plane.manifests: frozen dataclasses and loaders for ax's
Task, Workspace and Model kinds (ax.io/v1alpha1, ax's fields only) and the
Run kind (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs,
budget, serialize). Every violation raises ManifestError naming the kind,
metadata.name and field path: unknown fields at any depth, wrong apiVersion
or kind, a provider outside google/anthropic unless deepseek is declared via
the openultrasast.io/provider-extension annotation, a Run with a dependency
cycle, an input naming no task, and a Run task naming no loaded Task.
load_manifests reads multi-document YAML files and indexes by kind and name.
PyYAML becomes the one declared core dependency; the two zero-dependency
guards now pin exactly that, and the module audit registers the plane
package under the ai-service-plane spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
MeteredClient wraps a ChatClient, sums the provider's usage rows per call,
prices them from a Model manifest's parameters (cache-hit, input, output per
million), and raises BudgetExhausted before a call once the usd or calls
ceiling is reached. A 401/402 or "Insufficient Balance" failure becomes
AccountError with the provider's message. An unpriced model reports usd None,
never zero, and refuses a usd budget; summary() gives the spend block of a
task's summary.json.
The two plane modules are registered as spec-owned in the module audit (their
consumers land in tasks 3-4) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the main session: 90% of development tokens were tool I/O. Three
tools from the Portal pattern:
- read_guard.py: PreToolUse hook that denies a whole-file Read over 350 lines and
points to a ranged read or the bulk reader (pipe-tested: deny, range, small).
- bulk_read.py: answers a question about files with the cheap chat model, citing
path:line, under a spend ceiling.
- token_report.py: where a session's tokens go, from its transcript.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Exploratory, 7 of 17 cases, stopped when the account emptied:
- source-only scope: a model reading every product file puts the vulnerable
function among the candidates in 15/17 (ranker 1/17, vocabulary 2/17);
- per-file triage + per-file 6-step hunts + two-pass agreement: 12/15 candidate
cases on the validation set; 4/7 on the full run, two more in one pass of two;
- precision: 5 pgAdmin agreed findings read against source, 3 true (privileged),
2 false with self-contradicting rationales; agreement cannot catch an error
both passes make;
- cost: $0.009-0.022 per candidate for two passes.
Four instrument defects fixed on the way (model id, provider routing, JSON-mode
final turn, unbounded tool output) void the earlier hunter number. No gate is
met; qualification needs population v3 and a protocol that states the judge
stage, the pass rule and a config-family question.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- verify_batched.py --fixed asks the agreed candidates again at the fixed pin
(no triage); a function the fix removed is recorded as removed.
- score_batched.py applies protocol v2's matching to the batched output
(vulnerable pin, and the fixed pin where rechecked) and draws the precision
sample.
- A finding's site is anchored to the candidate operation asked about; the
model's own reported location is kept in `reported_at`. ReactPress's confirmed
findAll was reported at the controller file and failed to match its declared
site. The scorer anchors older records the same way.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".
On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
winml is triaged out as "constant": a CORS literal, which the reachability
question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
with several candidates, 2.3x cheaper per pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.
Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.
The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--declared-sites-only, --candidates-from <jsonl> and --results-suffix, so the
efficiency change can be validated on the 180 already-verified pgAdmin candidates
and the 15 declared sites without touching the main results.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs
$0.0139 per candidate, 90% of the spend, because the conversation is re-sent on
every turn and grows with each tool result.
- Context files now precede the question in the hunter's first message. DeepSeek
bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first,
every candidate asked about the same file (about 5 per file) broke the prefix on
its first line and paid for the file again. A test pins the order.
- The verifier runs each candidate at 3 steps and repeats it once at 6 only when
the hunt spent every step on tool calls without a verdict. On the seven declared
v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3.
- Per-candidate token usage (prompt, cached, completion, calls) and escalation
are recorded, and each case reports its cache share, so the next efficiency
decision is made on numbers.
Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1)
waits for account credit: DeepSeek returns 402 at the moment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The rerun hit DeepSeek HTTP 402 ("Insufficient Balance") after 180 pgAdmin
candidates. Every later call failed at $0, and the runner wrote sixteen cases as
complete with nothing asked, because an unverified candidate counted toward
completion. That is a clean-looking zero from an instrument that could not call
the model.
Now:
- 401/402 or "Insufficient Balance" stops the run at once, exit 2, nothing
recorded;
- only verified candidates count as done, so unverified ones are asked again on
resume;
- a case is complete only when every candidate was verified.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
FUXA failed every hunt with HTTP 400, and neither run could say why. The client
discarded the response body. With the body shown, the reason was "This model's
maximum context length...": FUXA ships minified bundles whose single lines run to
megabytes, and grep_repo returned a matched line whole.
- grep_repo and find_refs quote a matched line (240 characters), never the whole
line.
- Every tool result is capped at 16 x max_chars, with a truncation marker.
- OpenRouterChatClient errors carry the provider's response body (first 500
characters). That would also have exposed the rejected model id in one call
instead of eighteen days.
After the fix, FUXA candidates verify normally at $0.004-0.013 each. A test covers
the megabyte-line case.
verify_sinks.py: skip futures cancelled at the spend ceiling instead of reading
them (the run crashed with CancelledError).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Each grep_repo and find_refs call re-enumerated the tree (walk, ignore rules,
vendored checks) and re-read every file. The v2 sink verifier spent almost three
hours on its first case (pgAdmin, about 1,800 files) at 108% CPU with one open
connection. It was CPU-bound in these tools, not waiting on the model.
Source lines are now memoised per root, keyed by path and modification time. A hit
is re-validated file by file, because an edit inside a subdirectory does not touch
the root's mtime; a test covers one enumeration and a detected edit.
verify_sinks.py now runs candidates in worker processes, not threads, and appends
each candidate's result to a per-case log, so a stopped run resumes where it left
off. GoPay: 12 candidates in 14 s, the real site found.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Per candidate function, one bounded tool-hunter pass (custom prompt, the file as
context, 3 steps) asks whether attacker-controlled data reaches the operation
without a sufficient guard. It reports only with a stated source-to-sink path.
Hard spend ceiling, parallel workers; one run per pin; the fixed pin is rechecked
only for detected sites. Exploratory on v2.
Pilot: on GoPay, the one real site was reported in 3 of 4 runs and nothing else
out of 12 candidates.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Source-only selection on the v2 vulnerable pins, compared on the same cases:
- ranker top-20 entry points: the vulnerable file was in scope in 1/17;
- shipped-vocabulary sink-first: the declared site was a candidate in 2/17;
- model-classified sinks (deepseek-flash, no vocabulary): the declared site was a
candidate in 15/17, and a fix or site file in 16/17. Every chunk was classified;
$14.42.
Misses: User Registration (file in scope, not the function) and Sulu (JCR-SQL2
query builder). The cost of this reach is volume: 3-1037 candidate functions per
repository, 4,762 in total. The next stage must decide which candidates are fed by
attacker data without a guard. v2 is spent for tuning; this qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This asks DeepSeek, per product source file (in chunks of 400 numbered lines),
which functions perform the family's dangerous operation on non-constant data. It
is scored like sink_candidates.py: is the declared vulnerable function among the
candidates? A failed chunk counts as unclassified, never as "no sinks". The spend
ceiling is enforced across worker threads.
Pilot: GoPay found its declared site for $0.03. Estimated full v2 run: ~$14-18.
The maintainer raised the ceiling to $20.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is a source-only diagnostic: no engine and no model. At each v2 vulnerable
pin, every call matching the family's shipped sink vocabulary becomes a candidate,
attributed to its enclosing function.
- The declared vulnerable function is a candidate in 2/17 cases (Medplum, GoPay);
the ranker's top 20 managed 1/17.
- A fix-changed or site file is among the candidates in 8/17.
- Candidates per repository range from 2 to 904 functions.
The misses are vocabulary, not direction. Examples: wp_redirect, download_url,
spawn(shell:true), ky(), requests.Session().get, sympy parse_expr, Jinja
from_string().render, Mongo collection.find, JCR-SQL2 createQuery, TypeORM
andWhere template strings, and CORSMiddleware(allow_origins=["*"]) are not named.
v2 is spent for tuning, so this informs design and qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The run follows hunter-protocol-v2.md: deepseek-flash, the ranker's top 20
regions per pin, protocol v2 scoring, $8.88 spent.
- Recall: 1/17 (Wilson 0.01-0.27).
- Precision: 2/14 sampled (both true positives need a privileged role, one of
them GoPay's CVE).
- Fixed-side alerts: 1. Benign alerts: 16.
- Stability: 73% of findings differ between two identical runs.
The deciding observation: the ranker's top 20 contained the vulnerable file in 1
of 17 cases, and in that case (GoPay) the hunter detected it. With 4 tool steps
per hunt it cannot search its way to code it is not handed. The result therefore
says less about the model than about scope selection, which sits upstream of the
engine too.
Instrument failures, reported as failures: every FUXA pin and three Directorist
pins (DeepSeek HTTP 400 on every hunt; neither offered a region of its family).
Scorer fix: protocol v2 defines benign alerts and the precision pool over the
CASE FAMILY, but score_v2 pooled every family. The engine's dump emits only the
case family, so its recorded result (21adc40) is unchanged by this; rescoring
reproduces 3 benign alerts and 49 pooled findings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the deciding experiment of the strategy review, fixed before any v2 pin is
hunted.
- Hunter: the unchanged tool hunter (default prompt, tools and 4 steps) on
deepseek-flash.
- Scope: the ranker's top 20 regions for the case family, in four hunts. The model
never sees the fix diff or the declared sites.
- Families: a fixed keyword table maps the model's family to the protocol's three.
- Scoring: exactly protocol v2's scorer.
- Cost: a hard $10 ceiling.
hunt.py runs it and scores it (--score). A pilot on a v1 development case
(mongo-express) found the vulnerability at lib/bson.js:60, at $0.031 per pin,
alongside a lot of speculative noise.
Transport fix found by the pilot: the DeepSeek endpoint the hunter builds was
unpriced, so the cost meter read $0.00 for any number of calls. It is now priced.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was frozen at a75b9f6. 17 cases, 102 scans (vulnerable and fixed pins
twice each), scored exactly as pre-registered.
- Recall: 0/17 (Wilson 95% 0-0.18). No unstable detections.
- Precision: 0/17 sampled (0-0.18). Every sampled finding is false: regex
match/exec read as sinks, int casts, ORM statements, client-SDK code, the
process's own argv, a generated database name, a registered-URI check.
- Completion: 13206/108374 questions (12%).
- Fixed-side alerts: 0, trivially, since nothing was detected.
- Benign alerts: 3.
- Stability: 2 of 100 findings differ between runs.
The zero is not an instrument artefact: FUXA, GoPay, cct-studio, ReactPress, User
Registration and Directorist completed questions in the sink file itself and still
found nothing.
Instrument failures, reported as failures: LMDeploy (every taint batch failed on all
six pins) and Froxlor's two benign scans.
This record is committed before any analyzer change informed by v2.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any v2 case is scanned. It has two changes from v1, both chosen by
the maintainer from lessons of the v1 run:
- declared sites: a finding at a `path::function` the advisory names is credited,
as well as one on the fix's changed lines;
- two runs of the vulnerable and fixed pins: detection needs the finding in both
runs, a fixed-side alert in either counts, and the difference between runs is
reported as stability.
evaluate.py takes --population. v2 scans six pins per case and scores with
matches_v2 / score_v2. v1's scoring is unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The analyzer was not run. freeze.py read each case's source at both pins:
- both pins are commits in the repository and in the right order;
- the declared sink site's file is present at the vulnerable pin, and its declared
functions appear in it;
- the fix touches a declared site file (Budibase's sites reordered so the changed
file, queries.ts, comes first).
Benign controls:
- 12 are reviewed automatic proposals.
- 3 automatic proposals were rejected as not benign: qwed's merged a ReDoS fix,
ReactPress's corrected a SQL column whitelist, and Sulu's was a branch merge that
can carry the fix.
- Those three, plus cct-studio and Directorist (no automatic proposal), were chosen
by hand. Directorist's comes from before the vulnerable pin, because only 3
commits follow its fix.
freeze.py now accepts a recorded benign control instead of reporting that none was
proposed.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
v1 was evaluated and then used to retune the analyzer. A retune can only be
qualified on a population that did not inform it.
The candidates came from web research alone (research-2026-09-26.md):
- advisories were taken newest-first per language under fixed criteria, never by
how easy they look;
- 261 advisories were examined and 224 rejected, each with a reason;
- every SHA pair was verified through the GitHub API;
- no clone and no scan.
The maintainer selected the rule-based "balanced" set: 5 Python, 6 JS/TS, and 6 PHP
of which 4 are WordPress plugins; 10 injection, 6 destination and 1 config.
Each case declares the advisory's sink sites before any scan, for protocol v2.
freeze.py takes --population and writes freeze-vN.json. The reservation tests now
cover every population-v*.toml.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Scored exactly per the pre-registered protocol-v1.md, analyzer ebcc2cf,
44 scans on eleven untouched cases.
Recall 1/11 (95% CI 0.016-0.377): only Signal K's SSRF sink is detected.
Precision 1/12 on the full pooled sample: the other eleven findings are
guarded redirects (isSafeRelativeUrl, getSafeDestination), Host-header
redirects, constant SET NAMES queries, a config-file eval and an installer.
Four fixed-side false alerts (mongo-express bson.js, both wger sinks after
their guard was added) and one benign.
The dominant cause is coverage, not precision: at the vulnerable pin eight
of eleven cases completed ZERO questions and Saltcorn asked none, so most
targets were never decided. Recorded before any investigation or change.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Committed before any case is scanned: what is under test (the analyzer at the
runner's commit, per-case family only, four scans per case), how a finding
MATCHES a case (a fix-changed file, the case family, the named function or
within 10 lines of a fix hunk), detection / fixed-side / benign false alerts,
unanswered and instrument failures, the precision sampling rule (every third
pooled vulnerable-pin finding, or all if 15 or fewer), what is reported
against the M4 gates, and that no retune may be informed by the results.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Every reserved case verified with git and file reads only -- nothing
scanned: pins present and ordered, the sink present at the vulnerable pin,
the fix touching the sink (or LearnPress's declared source-side fix site),
and the sink file hashed at both pins (benchmarks/independent/freeze.py,
record freeze-v1.json).
Each case now carries a benign control from its own history, reviewed by
diff. The mechanical keyword filter was not enough: it proposed an access
restriction, a filter check, API-token scopes, a login change and a
TLS-verification option, all rejected; Ultimate Member and alerta got
hand-picked controls (a WPCS reformat; a dependency change with both sides
still vulnerable). The guard test requires a benign pair per case once
frozen.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Eleven cases chosen by the maintainer from fifteen researched by advisory
and fix diff only -- nothing cloned, nothing scanned, none among the 262
repositories the benchmarks reference -- plus the earlier Ghost
reservation. Four are PHP repositories that ship JavaScript (two WordPress
plugins), satisfying the multi-language requirement; families are limited
to the v1 scope (injection, untrusted destination, permissive config).
The development subjects shaped every rule of the last week and can no
longer measure whether those rules generalise; this population can, until
something scans it. A guard test fails the moment a reserved repository is
referenced by a recipe, catalog, manifest or measurement outside
benchmarks/independent/. Freezing (pins and sinks verified by reading
source, a benign change per case) is the next step.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
