Agent Substrate's golden-snapshot boot runs the same image with the same task env,
so the runner no longer starts its command by itself: after the workspaces are
ready it waits for POST /ousast/v1/start (run/task checked, once per boot, refused
after a delivered run). The reconciler sends it through svc/atenet-router with
ate-target-actor, which never addresses the golden actor (plane/router.py), and the
bound Model's secretKey variable travels only in that body, into the child's env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The live cluster ran the task command twice: in the actor and in Substrate's
golden-snapshot boot, which gets the same image and task env. Requirement 4.5
gates the command on a router-addressed start request; 4.6 carries provider
credentials only in that request.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
On the live cluster ax hands AX_TASK_YAML with metadata.creationTimestamp and a
status block; the strict authoring schema rejected it, readiness never turned
green and every resume timed out. Other unknown fields still fail. up.sh retries
the first resume while the golden snapshot builds.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The first live run on the local ax cluster and ax's source (docs/runner.md,
docs/sandbox.md, internal/controller/reconciler.go, internal/server/server.go)
showed a lifecycle the task-3/4 code did not model:
- `ax apply` creates a Task Suspended (Ready=False, TaskSuspended); it runs only
after `ax resume task`. The first resume of a new image can fail with
DeadlineExceeded while Agent Substrate builds the golden snapshot, and ax's
server then stores phase Failed; a later resume succeeds.
- ax has no Completed phase and never reads a command's exit back.
- The runner is PID 1 and must keep serving after the command exits; stop and
suspend deliver SIGTERM, to be forwarded to the command's process group.
Our runner imported the task in-process and exited, which stopped the
container ("Container guest is stopped") and made every resume time out.
- ax's controller probes /readyz?check=workspace; the query string made our
handler answer 404.
Runner (Req 4.1-4.4): spec.command runs as a child
`python -m openultrasast.plane.tasks.<name>` in its own process group, cwd the
first workspace, AX_METADATA_URL set; ax's metadata paths are served. After the
child exits the runner reads summary.json (a crash without one becomes failed
with the exit code and stderr tail), delivers the output directory, writes
/workspace/.ousast-state/<task>.done.json and serves until SIGTERM (forwarded
to the group, SIGKILL after OUSAST_TERM_GRACE, exit 0). A boot with the marker
never reruns the command; a failed delivery is retried alone. Workspaces are
prepared on the first boot only. AX_RUNNER_EXIT_AFTER_COMMAND=1 is the
unit-test escape.
Reconciler (Req 2.2, 2.3, 3.1-3.3): `ax resume task` after apply, retrying
DeadlineExceeded/Unavailable with backoff up to OUSAST_RESUME_TIMEOUT (900 s);
completion is the artifact delivery; Failed after a resume went through, a
vanished task, or no delivery within OUSAST_TASK_TIMEOUT (7200 s) fail the
task with ax's condition message; `ax delete task` follows. Ax gains resume()
and phase() returns the condition message. doctor moves to plane/doctor.py
(tasks.md task 0 allows the sibling) to keep reconciler.py under 500 lines.
Tests: the fake ax follows the real lifecycle (Suspended, a failing first
resume that leaves Failed, Running forever, delivery only after a resume went
through), plus Failed-with-message, vanish, resume-timeout and task-timeout
scenarios; runner tests cover the child's cwd/group/metadata URL, stay-up and
SIGTERM in a real subprocess, SIGKILL of a SIGTERM-ignoring group, and the
marker. The opt-in live test now checks resume, Running, suspend and a second
resume instead of a Completed phase. design.md and tasks.md state the
lifecycle; the module audit manifest counts plane.doctor.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Model deepseek-flash (declared provider extension, endpoint prices in
parameters, secretKey naming the DEEPSEEK_API_KEY secret), digest-pinned
Task templates repo-facts / verify / agree, and a generator
(plane/generate.py, `ousast plane workspaces --validation-set ...`) that
emits per case the vulnerable-pin Workspace, a files-only inputs Workspace
(recorded candidates after the recorded triage, declared sites, triage),
per-case Tasks and one Run validation-46: facts -> verify a, verify b ->
agree for the 15 cases, 46 candidates (43 kept), budgets 1.5x the recorded
per-pass share. A Run entry cannot add a binding or env, and Run.inputs
names only task artifacts, so the case is fixed by per-case Tasks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Substrate installs no WorkerPool (resume failed: no free workers) and rejects
images not pinned by digest; ax's default runner image on gcr.io needs
credentials, so the smoke task runs our runner (repo-facts over a one-file
workspace). A release ax CLI skewed from the server (UpdateTask unimplemented).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirement 1.1 (Task/Workspace/Model use only fields ax documents) and
Requirement 1.4 (no ax field repurposed) were violated: manifests accepted
metadata.namespace and a Workspace git `commit`, and the reconciler dumped
every git field into what `ax apply` receives.
Facts from ax's source (pkg/apis/v1alpha1/ax.proto, internal/workspace/setup.go):
- ObjectMeta is `name` and `atespace` (plus server-set creation_timestamp);
no `namespace`, no `annotations`.
- GitRepo is `name`, `repo`, `branch`, `dir`, `depth`; no `commit`.
- The API server decodes with protojson.UnmarshalOptions{}: an unknown field
is an error at `ax apply` (confirmed on the live cluster: a rendered
Workspace+Model applied cleanly; the same Workspace with `commit` was
refused with `unknown field "commit"`).
- ValidateModel checks only metadata; any spec.provider string is accepted.
Changes:
- manifests: metadata takes `name`, `atespace`, `annotations` (our extension,
never sent to ax); git entries take exactly ax's five fields (depth int);
`commit` in spec.git is rejected with a pointer to the new Workspace
annotation `openultrasast.io/git-commits: "<git name>=<40-hex>,..."`,
validated for unknown git names and non-hex shas.
- reconciler: rendered docs carry only ax fields; each bound Workspace's pins
reach the Task as env OUSAST_GIT_PINS = {"<workspace>/<git name>": "<sha>"}.
A test walks every rendered doc against ax's field lists.
- runner: pins come from OUSAST_GIT_PINS; git is fetched as setup.go does
(init/remote/fetch with --depth when depth > 0/checkout FETCH_HEAD, default
branch main, cloneDestination's dir and placeholder-name rules), then the
pin is checked out. A pinned entry without a branch fetches the commit.
- workspaces: emits the annotation, `dir: repo` and `depth: 1` instead of
`commit`, so generated case workspaces keep <path>/repo at depth 1.
- design.md: ax field names verbatim; pins and model binding as stripped
openultrasast.io/* annotations carried to the task by env.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kubectl prints the sentence as one whitespace-split row, which passed the
ready check and reported an absent ax controller as "1 pods ready". Rows must
carry a READY column; an empty namespace fails. Regression test added.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
ai-service-plane task 3, Requirements 4.1-4.4 (with 2.2 and 2.3 honoured by the
exit-code contract):
- src/openultrasast/plane/runner.py: the ax-task-runner entrypoint. Reads the Task
from AX_TASK_YAML and the bound Workspaces from AX_WORKSPACES_YAML (task-1 loaders),
serves /healthz and /readyz on port 80 (AX_RUNNER_PORT; 0 picks a free port,
AX_RUNNER_HTTP=0 disables it for unit tests), materialises each workspace at its
bound path (spec.files verbatim; spec.git at <path>/<name>, exported from the
OUSAST_CASE_CACHE checkout whose origin matches, else a shallow clone at the pin;
a repo already present on a resumed volume is kept) and only then answers 200 on
/readyz. Runs spec.command by importing openultrasast.plane.tasks.<name> with the
remaining elements as argv; sets OUSAST_WORKSPACE_DIR from the first binding when
the Task's env leaves it unset. A crash writes summary.json with status failed and
the traceback. When OUSAST_ARTIFACT_URL is set, the whole OUSAST_OUTPUT_DIR is
POSTed as an uncompressed tar (Content-Type application/x-tar, X-Ousast-Task,
X-Ousast-Run) with three attempts and backoff; a failed delivery rewrites
summary.json to failed and exits 2. Exit codes: done 0, failed 2, unfinished 3.
- plane/Dockerfile.runner: python:3.12-slim + git, `pip install .` (no extras), a
shell wrapper at /usr/local/bin/ax-task-runner exec'ing the module; built and
loaded into kind by ops/ax/up.sh, not by the tests.
- tests/test_plane_runner.py (+ tests/plane_stub_task.py registered as the task
module): env contract on a files workspace, readyz 503 then 200, delivery to a
local receiver with the headers and tar checked, delivery failure -> failed and
exit 2 after three attempts, crash -> failed with traceback, exit codes per
status, case-cache export at the pin, shallow-clone fallback, resume keeps the
checkout.
- model/audit.py: SPEC_OWNED row for plane.runner; module-audit measurement
regenerated (122 modules, no orphans, note kept).
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 3.1, 3.2, 3.3, 3.4, 3.5, 2.2, 7.1, 7.3 of ai-service-plane.
- plane/reconciler.py (492 lines, test-counted): a Run's tasks submitted to ax
in dependency order through the injectable `ax` CLI (apply / get task /
delete task), serialize labels never overlapping, --workers, state.json with
pending|running|suspended|done|failed|unfinished, a PID lock per run, done
tasks skipped on rerun, unfinished tasks holding their dependants, a failed
task halting the run. Budgets, inputs, output dir, artifact URL and the bound
Model (annotation openultrasast.io/model; identity and priced parameters) go
to the task as env in the rendered Task; consumed artifacts travel as files
of a generated Workspace bound under /workspace. A threaded HTTP receiver
takes each task's tar delivery, validating run/task and rejecting traversal.
Outcome comes from the delivered summary.json (done only when every unit
finished), never from ax's phase alone. status prints the token attribution
table per task and Model from the summaries' usage fields (n/a, never an
estimate) and writes attribution.json; doctor runs the four host checks.
- plane/workspaces.py: one Workspace manifest per case pin of a population.
- CLI: ousast plane run|status|doctor|workspaces.
- tests/test_plane_reconciler.py: a fake ax executable drives the DAG;
tests/test_plane_ax_live.py (marker ax) applies ops/ax/smoke-task.yaml on a
live cluster and skips unless doctor passes.
- Module audit rows and regenerated manifest.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Requirements 5.3, 2.1, 2.2, 2.3, 7.2 of ai-service-plane.
- plane/tasks/verify.py: the batched hunt of verify_batched.py moved into
the package (system prompt, one hunt per file, up to 6 candidates, 6
steps, candidate-anchored sites). New against the reference: each
candidate's known callers from facts.json in the prompt (capped at 8),
tool turns and usage per hunt in units.jsonl, per-unit resume (rows
with "error" are re-asked), pass label from env, MeteredClient wrapper:
BudgetExhausted -> unfinished (exit 3), AccountError -> failed with the
provider message (exit 2); summary.json is written last, names the bound
Model and the summed usage, and is done only when every unit finished.
- plane/tasks/agree.py: verify_batched.agreement over two pass outputs,
agreed.json / disputed.json, per-candidate cost and turns, declared-site
match via benchmarks/independent/evaluate.matches_v2 loaded by file;
metrics block in summary.json; no model calls.
- tests/test_plane_verify.py: a scripted usage-recording hunter client;
callers line, resume, 402, budget stop and resume, model and summed
usage, unpriced usd None, both main() env contracts, agreement metrics.
- model/audit.py + module-audit manifest: SPEC_OWNED rows for plane.tasks,
plane.tasks.verify, plane.tasks.agree; manifest regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
plane/tasks/repo_facts.py computes, from source text only and without a
model or git, the product files of a workspace (tests, vendored trees,
node_modules, dist/build, minified bundles and dot-directories excluded via
the layout facts plus the model_sinks.py filter), the functions each file
declares (Python, PHP, JavaScript/TypeScript, Java, Go, Ruby declaration
patterns extended from sink_candidates.py) and, per candidate function, the
call sites in other product files with path, 1-based line and enclosing
function, capped at 40 and sorted (Requirement 5.1). Output is byte-identical
across runs over the same tree (Requirement 5.2).
Under the task contract the unit is one file: a row goes to units.jsonl as
each file finishes and a restart reuses the rows computed for the same
candidate set (Requirement 2.2); facts.json is assembled from the rows and
summary.json is written last, refusing `done` unless every unit finished.
The entrypoint reads OUSAST_WORKSPACE_DIR, OUSAST_OUTPUT_DIR and optional
OUSAST_INPUT_CANDIDATES only (Requirement 2.1) and exits 0 on done, 2 on
failed with the traceback recorded in summary.json.
tests/test_plane_repo_facts.py: a three-file tree with a tests/ file, a
vendored file, a node_modules bundle and a dot-directory; cross-file callers
with line and enclosing function, same-file and declaration lines excluded,
the 40 cap, byte-identical second run, resume that recomputes only the
missing unit, a crash that leaves `failed` with the unit counts and resumes
to `done`, and the declaration patterns per language.
The two new modules are registered as spec-owned in the module audit (their
importer, the runner, lands in task 3) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
kind cluster + registry, Agent Substrate, ax control plane via ko, one files-only
smoke Task; helpers run from their own checkout because they resolve paths from
the git toplevel. Footprint and README follow once the cluster is up.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Add openultrasast.plane.manifests: frozen dataclasses and loaders for ax's
Task, Workspace and Model kinds (ax.io/v1alpha1, ax's fields only) and the
Run kind (openultrasast.io/v1alpha1: tasks, dependsOn, inputs, outputs,
budget, serialize). Every violation raises ManifestError naming the kind,
metadata.name and field path: unknown fields at any depth, wrong apiVersion
or kind, a provider outside google/anthropic unless deepseek is declared via
the openultrasast.io/provider-extension annotation, a Run with a dependency
cycle, an input naming no task, and a Run task naming no loaded Task.
load_manifests reads multi-document YAML files and indexes by kind and name.
PyYAML becomes the one declared core dependency; the two zero-dependency
guards now pin exactly that, and the module audit registers the plane
package under the ai-service-plane spec.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Maintainer decisions 2026-09-29: ax (kind + Agent Substrate + ax controller on
this host) is the only executor; the Run layer carries dependencies, artifacts,
budgets and attribution until ax has them. Requirement 7 declares routing per
Task (one Model per Task, no runtime switching) and a token attribution table
from provider usage fields. Task 0 (bring-up, doctor), runner artifact delivery,
ax-backed reconciler with a fake-ax test seam. harnessx-removal brief records
the HarnessX inventory; removal is gated on the increment's measurement.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
MeteredClient wraps a ChatClient, sums the provider's usage rows per call,
prices them from a Model manifest's parameters (cache-hit, input, output per
million), and raises BudgetExhausted before a call once the usd or calls
ceiling is reached. A 401/402 or "Insufficient Balance" failure becomes
AccountError with the provider's message. An unpriced model reports usd None,
never zero, and refuses a usd budget; summary() gives the spend block of a
task's summary.json.
The two plane modules are registered as spec-owned in the module audit (their
consumers land in tasks 3-4) and the audit manifest is regenerated.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Nine tasks in dependency order: manifests, budget metering, runner contract,
reconciler and CLI, repo-facts, verify and agree, the increment's manifests, the
measurement against the recorded batched check, handover.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Package src/openultrasast/plane/: manifests (ax kinds + Run), a reconciler under
500 lines with persisted status and resume, an ax-contract runner, budget metering,
and task entrypoints. First increment: repo-facts (source-only callers, no model),
verify (the batched hunt with callers in the prompt, per-unit resume), agree.
Measurement on the 46-candidate set against the recorded batched check.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Six requirements from the brief: ax kinds plus a Run kind; isolated, idempotent,
resumable tasks with metered budgets; a local reconciler under 500 lines with no
pipeline logic; ax's runner contract; repo-facts as the memory layer; and a
measured first increment (repo-facts + verify on the 46-candidate set: at least 16
of 20 declared sites, under $0.022 per candidate for two passes).
Phase: requirements, not yet approved.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Measured on the main session: 90% of development tokens were tool I/O. Three
tools from the Portal pattern:
- read_guard.py: PreToolUse hook that denies a whole-file Read over 350 lines and
points to a ranged read or the bulk reader (pipe-tested: deny, range, small).
- bulk_read.py: answers a question about files with the cheap chat model, citing
path:line, under a spend ceiling.
- token_report.py: where a session's tokens go, from its transcript.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Adopt the design of google/ax (Task / Workspace / Model manifests, a reconciler,
isolated resumable execution, a runner contract), not yet its software: it needs
Kubernetes and Agent Substrate, lists only Google and Anthropic providers, and has
no pipelines, artifacts or budgets. We add a thin Run layer for those.
Each pipeline concern (classify, triage, repo-facts, verify, judge, recheck, score)
becomes a declared task with its own inputs, outputs and budget. That bounds the
tokens per task and lets each be developed alone. Decisions listed for the
maintainer.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Exploratory, 7 of 17 cases, stopped when the account emptied:
- source-only scope: a model reading every product file puts the vulnerable
function among the candidates in 15/17 (ranker 1/17, vocabulary 2/17);
- per-file triage + per-file 6-step hunts + two-pass agreement: 12/15 candidate
cases on the validation set; 4/7 on the full run, two more in one pass of two;
- precision: 5 pgAdmin agreed findings read against source, 3 true (privileged),
2 false with self-contradicting rationales; agreement cannot catch an error
both passes make;
- cost: $0.009-0.022 per candidate for two passes.
Four instrument defects fixed on the way (model id, provider routing, JSON-mode
final turn, unbounded tool output) void the earlier hunter number. No gate is
met; qualification needs population v3 and a protocol that states the judge
stage, the pass rule and a config-family question.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
- verify_batched.py --fixed asks the agreed candidates again at the fixed pin
(no triage); a function the fix removed is recorded as removed.
- score_batched.py applies protocol v2's matching to the batched output
(vulnerable pin, and the fixed pin where rechecked) and draws the precision
sample.
- A finding's site is anchored to the candidate operation asked about; the
model's own reported location is kept in `reported_at`. ReactPress's confirmed
findAll was reported at the controller file and failed to match its declared
site. The scorer anchors older records the same way.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
verify_batched.py replaces the per-candidate verifier's design:
- triage per file (no tools): where does the operation's data originate;
internal/constant candidates are not verified;
- one 6-step hunt per file for up to 6 candidates, a verdict per operation;
- two independent passes; only agreement is reported, the rest is "disputed".
On the 46-candidate validation set ($1.01):
- declared sites: 16 of 20 agreed, 12 of 15 cases (the per-candidate design
confirmed 11). Budibase and Froxlor are new; Unleash dropped to one pass of two.
winml is triaged out as "constant": a CORS literal, which the reachability
question does not fit.
- the 26 pgAdmin candidates that flickered between single runs: 2 triaged out,
9 agreed in both passes, 4 disputed, 13 quiet.
- $0.022 per candidate for two passes vs $0.033 under the old design; on a file
with several candidates, 2.3x cheaper per pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Declared vulnerable sites: 11 of the 15 that were candidates are confirmed (8 at 3
steps, 3 more at 6). Missed: winml (a CORS config; the reachability question does
not fit), FUXA, Budibase, Froxlor.
Restart-based escalation is strictly worse than a straight 6-step budget: 88% of
pgAdmin candidates used every step, so they paid 3 + 6 ($0.0201 per candidate vs
$0.0139 baseline). Defaults are now 6 steps, no escalation. The cache order change
did not lower the per-call cost ($0.002): the conversation prefix was already
cached; 56% of prompt tokens are served from cache.
The larger finding: on 180 pgAdmin candidates verified twice, 15 and 13 were
flagged with only 2 in common. Verdicts on borderline candidates flip run to run,
while the declared sites were found consistently. Precision needs agreement
across runs or a second judgement, not a single pass.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
--declared-sites-only, --candidates-from <jsonl> and --results-suffix, so the
efficiency change can be validated on the 180 already-verified pgAdmin candidates
and the 15 declared sites without touching the main results.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Token efficiency, measured on 180 verified pgAdmin candidates: stage 2 costs
$0.0139 per candidate, 90% of the spend, because the conversation is re-sent on
every turn and grows with each tool result.
- Context files now precede the question in the hunter's first message. DeepSeek
bills a cached prefix at 1/30 of a fresh one, by prefix; with the question first,
every candidate asked about the same file (about 5 per file) broke the prefix on
its first line and paid for the file again. A test pins the order.
- The verifier runs each candidate at 3 steps and repeats it once at 6 only when
the hunt spent every step on tool calls without a verdict. On the seven declared
v2 sites, 3 steps found 5 and 8 steps found 6; most candidates conclude in 3.
- Per-candidate token usage (prompt, cached, completion, calls) and escalation
are recorded, and each case reports its cache share, so the next efficiency
decision is made on numbers.
Validation on the 180 pgAdmin candidates plus the 22 known sites (about $1)
waits for account credit: DeepSeek returns 402 at the moment.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The rerun hit DeepSeek HTTP 402 ("Insufficient Balance") after 180 pgAdmin
candidates. Every later call failed at $0, and the runner wrote sixteen cases as
complete with nothing asked, because an unverified candidate counted toward
completion. That is a clean-looking zero from an instrument that could not call
the model.
Now:
- 401/402 or "Insufficient Balance" stops the run at once, exit 2, nothing
recorded;
- only verified candidates count as done, so unverified ones are asked again on
resume;
- a case is complete only when every candidate was verified.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
FUXA failed every hunt with HTTP 400, and neither run could say why. The client
discarded the response body. With the body shown, the reason was "This model's
maximum context length...": FUXA ships minified bundles whose single lines run to
megabytes, and grep_repo returned a matched line whole.
- grep_repo and find_refs quote a matched line (240 characters), never the whole
line.
- Every tool result is capped at 16 x max_chars, with a truncation marker.
- OpenRouterChatClient errors carry the provider's response body (first 500
characters). That would also have exposed the rejected model id in one call
instead of eighteen days.
After the fix, FUXA candidates verify normally at $0.004-0.013 each. A test covers
the megabyte-line case.
verify_sinks.py: skip futures cancelled at the spend ceiling instead of reading
them (the run crashed with CancelledError).
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The final-answer turn requested the provider's JSON mode with the tools still
attached. DeepSeek (deepseek-flash) then answered with the format itself
(`{"type": "json_object"}`), wrapped the array (`{"type": "json_object",
"value": [...]}`), or emitted its tool-call markup as text. A hunt that had found
the vulnerability reported nothing.
Measured on the v2 sink verifier with the declared vulnerable functions: 0 of 7
confirmed in JSON mode, 5 of 7 asked plainly (6 of 7 at 8 steps). The model named
Medplum's actual redirect-URI prefix-match bug, LMDeploy's redirect bypass of
_is_safe_url, and ContextForge's unsandboxed Jinja2.
The final turn is now plain text without tools. The parser also reads a
`{"value": [...]}` wrapper. Two tests that pinned JSON mode now pin the plain
turn, and a new test reproduces the wrapped answer.
Invalidates as measurements: the hunter experiment on v2 (c4a0bd8) and the first
sink-verifier run. Both used this turn.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Each grep_repo and find_refs call re-enumerated the tree (walk, ignore rules,
vendored checks) and re-read every file. The v2 sink verifier spent almost three
hours on its first case (pgAdmin, about 1,800 files) at 108% CPU with one open
connection. It was CPU-bound in these tools, not waiting on the model.
Source lines are now memoised per root, keyed by path and modification time. A hit
is re-validated file by file, because an edit inside a subdirectory does not touch
the root's mtime; a test covers one enumeration and a detected edit.
verify_sinks.py now runs candidates in worker processes, not threads, and appends
each candidate's result to a per-case log, so a stopped run resumes where it left
off. GoPay: 12 candidates in 14 s, the real site found.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Per candidate function, one bounded tool-hunter pass (custom prompt, the file as
context, 3 steps) asks whether attacker-controlled data reaches the operation
without a sufficient guard. It reports only with a stated source-to-sink path.
Hard spend ceiling, parallel workers; one run per pin; the fixed pin is rechecked
only for detected sites. Exploratory on v2.
Pilot: on GoPay, the one real site was reported in 3 of 4 runs and nothing else
out of 12 candidates.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
Source-only selection on the v2 vulnerable pins, compared on the same cases:
- ranker top-20 entry points: the vulnerable file was in scope in 1/17;
- shipped-vocabulary sink-first: the declared site was a candidate in 2/17;
- model-classified sinks (deepseek-flash, no vocabulary): the declared site was a
candidate in 15/17, and a fix or site file in 16/17. Every chunk was classified;
$14.42.
Misses: User Registration (file in scope, not the function) and Sulu (JCR-SQL2
query builder). The cost of this reach is volume: 3-1037 candidate functions per
repository, 4,762 in total. The next stage must decide which candidates are fed by
attacker data without a guard. v2 is spent for tuning; this qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This asks DeepSeek, per product source file (in chunks of 400 numbered lines),
which functions perform the family's dangerous operation on non-constant data. It
is scored like sink_candidates.py: is the declared vulnerable function among the
candidates? A failed chunk counts as unclassified, never as "no sinks". The spend
ceiling is enforced across worker threads.
Pilot: GoPay found its declared site for $0.03. Estimated full v2 run: ~$14-18.
The maintainer raised the ceiling to $20.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is a source-only diagnostic: no engine and no model. At each v2 vulnerable
pin, every call matching the family's shipped sink vocabulary becomes a candidate,
attributed to its enclosing function.
- The declared vulnerable function is a candidate in 2/17 cases (Medplum, GoPay);
the ranker's top 20 managed 1/17.
- A fix-changed or site file is among the candidates in 8/17.
- Candidates per repository range from 2 to 904 functions.
The misses are vocabulary, not direction. Examples: wp_redirect, download_url,
spawn(shell:true), ky(), requests.Session().get, sympy parse_expr, Jinja
from_string().render, Mongo collection.find, JCR-SQL2 createQuery, TypeORM
andWhere template strings, and CORSMiddleware(allow_origins=["*"]) are not named.
v2 is spent for tuning, so this informs design and qualifies nothing.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
The run follows hunter-protocol-v2.md: deepseek-flash, the ranker's top 20
regions per pin, protocol v2 scoring, $8.88 spent.
- Recall: 1/17 (Wilson 0.01-0.27).
- Precision: 2/14 sampled (both true positives need a privileged role, one of
them GoPay's CVE).
- Fixed-side alerts: 1. Benign alerts: 16.
- Stability: 73% of findings differ between two identical runs.
The deciding observation: the ranker's top 20 contained the vulnerable file in 1
of 17 cases, and in that case (GoPay) the hunter detected it. With 4 tool steps
per hunt it cannot search its way to code it is not handed. The result therefore
says less about the model than about scope selection, which sits upstream of the
engine too.
Instrument failures, reported as failures: every FUXA pin and three Directorist
pins (DeepSeek HTTP 400 on every hunt; neither offered a region of its family).
Scorer fix: protocol v2 defines benign alerts and the precision pool over the
CASE FAMILY, but score_v2 pooled every family. The engine's dump emits only the
case family, so its recorded result (21adc40) is unchanged by this; rescoring
reproduces 3 benign alerts and 49 pooled findings.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This is the deciding experiment of the strategy review, fixed before any v2 pin is
hunted.
- Hunter: the unchanged tool hunter (default prompt, tools and 4 steps) on
deepseek-flash.
- Scope: the ranker's top 20 regions for the case family, in four hunts. The model
never sees the fix diff or the declared sites.
- Families: a fixed keyword table maps the model's family to the protocol's three.
- Scoring: exactly protocol v2's scorer.
- Cost: a hard $10 ceiling.
hunt.py runs it and scores it (--score). A pilot on a v1 development case
(mongo-express) found the vulnerability at lib/bson.js:60, at $0.031 per pin,
alongside a lot of speculative noise.
Transport fix found by the pilot: the DeepSeek endpoint the hunter builds was
unpriced, so the cost meter read $0.00 for any number of calls. It is now priced.
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
