From 6d601c14647d1464b070a5b19184e8099c3650af Mon Sep 17 00:00:00 2001 From: Marius Ciepluch <11855163+norandom@users.noreply.github.com> Date: Fri, 2 Oct 2026 10:42:17 +0200 Subject: [PATCH] docs: token ergonomics -- runtime routing/budgets/caches and the development-side tooling One page, two halves with a Mermaid diagram each. A: the plane's token economy (one Model per Task with a metered usd/calls budget, attribution table, credentials only in the start request, repo-facts and content-addressed reuse, replay at $0, triage-hunt-tie-break order, prefix-cache prompt order) with every cost quoted from plane-increment-1/2.json, the injection-slice and harvest records, and an honest not-yet list (no token budget in ax, triage applied not run, k and contrast fixed outside experiments). B: the development-side measurement (tool I/O 85.5%, previously 90%) and the benchmarks/dev remedies: read_guard hook, bulk_read portal, token_report, plus the practices. Linked from the nav, index, memory-and-detection and the README docs list. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo --- README.md | 2 + docs/index.md | 2 + docs/memory-and-detection.md | 3 +- docs/token-ergonomics.md | 234 +++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 5 files changed, 241 insertions(+), 1 deletion(-) create mode 100644 docs/token-ergonomics.md diff --git a/README.md b/README.md index 77fb0dd..b9b63aa 100644 --- a/README.md +++ b/README.md @@ -229,6 +229,8 @@ Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra pair corpora and the plane increment results. - [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine. - [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening. +- [docs/token-ergonomics.md](docs/token-ergonomics.md): model spend at runtime (budgets, reuse, + measured costs) and in coding sessions. - [docs/examples.md](docs/examples.md): end-to-end walkthroughs. ## Development diff --git a/docs/index.md b/docs/index.md index d6f7bce..c5091a6 100644 --- a/docs/index.md +++ b/docs/index.md @@ -49,6 +49,8 @@ flowchart LR - Understand a finding's evidence and score: [Architecture](architecture.md). - Operate the plane and its store: [ax on this host](ops/ax/README.md) and [RustFS setup](rustfs.md). +- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions, + and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md). - Trust boundaries and hardening: [Threat model](threat-model.md). Every number in these pages is quoted from a committed record whose path is given next to it. diff --git a/docs/memory-and-detection.md b/docs/memory-and-detection.md index 77ee9d7..fcc4cdd 100644 --- a/docs/memory-and-detection.md +++ b/docs/memory-and-detection.md @@ -141,7 +141,8 @@ Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/reco only). Family **injection**; memory of 724 static examples from all families; input profile v1; framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped 5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap -intervals. +intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at +$0) is on [Token ergonomics](token-ergonomics.md#measured-costs). | Measure | Value | What it means | | --- | --- | --- | diff --git a/docs/token-ergonomics.md b/docs/token-ergonomics.md new file mode 100644 index 0000000..fbf2990 --- /dev/null +++ b/docs/token-ergonomics.md @@ -0,0 +1,234 @@ +# Token ergonomics + +Model tokens are the one running cost of this project, and they are spent in two places: by the +**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers' +own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy +is a routing concern. ax can handle that and structure our services in small components." So the +runtime half is routing, one Model per Task with its own budget, and reuse of what was already +paid for; the development half is tooling that keeps bulk file contents out of the session. + +Every number on this page comes from a committed record, named next to it. Records are counts +only; none holds code, prompts or responses. + +## A. Runtime: the plane's token economy + +```mermaid +flowchart LR + subgraph budget ["Budget and attribution"] + start["start request (credentials only here)"] --> task["Task: one Model, budget usd and calls"] + task --> meter["metered client refuses the next call at the ceiling"] + meter --> summary["summary.json: calls, tokens, usd"] + summary --> status["ousast plane status: attribution.json per task and per Model"] + end + subgraph reuse ["Pay once, reuse"] + facts["repo-facts, model-free"] -- "same repo, pin, candidates, image" --> seeded["seeded: calls 0, no usd"] + blobs["excerpts, embeddings, responses, programs by sha256"] -- "replay" --> zero["rerun at $0"] + end + subgraph order ["Cheap before expensive"] + triage["recorded triage per file"] --> hunt["one hunt per file, up to 6 candidates"] + hunt --> agree["agree: a and b"] + agree -- "disputed only" --> tiebreak["pass c, 2-of-3"] + end +``` + +### One Model per Task, one budget per Task + +A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}` +([The plane on ax](plane.md)). The budget is enforced by the metered client +(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and +raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call +that crosses the ceiling completes and the next one is refused. The task then ends `unfinished` +and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or +"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further. +Usage is attributed per call: the wrapped client appends one usage row per call +(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when +the call raises, and they are priced by the Model's price list. An unpriced Model cannot run +under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run +with `{usd: 0, calls: 0}`, so a stray call fails loudly. + +`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`, +`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a +`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit +rows. The same table is written as `attribution.json` in the run directory +(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials +reach a task only in the start request: `router.py` reads the variable the Model's +`secretKey.key` names, the runner keeps the value in memory, passes it to the command's +environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task +or a file. + +### Pay once, reuse + +- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no + model, no git) and its `facts.json` is byte-identical across runs over the same tree. The + memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image + digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task + carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with + `calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin, + candidate set or image finds nothing and recomputes. +- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as + `excerpts/.txt`; an embedding lives at + `embeddings//.json` and "an excerpt is embedded once; a rerun makes no + call" (`embeddings.py`). Model responses are cached under the sha256 of + `{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a + compiled program is `programs/.json` with the digests of the memory + snapshot and of the response cache it was built from (`compile.py`). +- **Replay is free.** With no client, the response cache replays at $0 and a miss raises + `ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run + replay-only on the committed code: 550 responses replayed, $0, identical overall and + calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`, + `replay_check`). + +### Cheap before expensive + +- **Triage first.** Verification runs only over the candidates the recorded triage kept; the + plane applies that recording without a model call and prices it as a separate line + (`recorded_triage_usd`), so every per-candidate cost below is reported both with and without + it (`src/openultrasast/plane/tasks/agree.py`). +- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6` + candidates, with the candidates' known callers (capped at 8) in the prompt + (`src/openultrasast/plane/tasks/verify.py`, `units_of()`). +- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is + started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done` + with no model call, and the final verdict is 2-of-3 ("pass c alone never decides", + `agree.py`). +- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix + (instruction and fixed demonstrations, identical for every candidate so the provider's + prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`). + Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache + for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice + record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix + too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits + (`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`, + `detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)"). + +### Measured costs + +| Measurement | Spend | Record | +| --- | --- | --- | +| Plane increment 2: verify a, b and the tie-break on the 46-candidate validation set | $0.0205 per candidate over the 46 including the recorded triage, against the $0.022 reference (`gate.cost_per_candidate_with_recorded_triage_usd`); totals: plane $0.8567, recorded triage $0.0872, with triage $0.9439, reference $1.0123 (`detail.usd`); the tie-break asked 10 disputed candidates and flipped 5 to agreed | `benchmarks/independent/plane-increment-2.json` | +| Plane increment 1: verify a and b alone | 357 calls, 54 hunts at 5.37 turns per hunt, $0.7471 plus $0.0872 recorded triage (`detail`) | `benchmarks/independent/plane-increment-1.json` | +| Decision engine, injection slice: evaluation at k = 5 | $0.003215 per candidate (`spend.cost_per_candidate_evaluate_usd`); metered at list price: canary $0.010158, compile $0.429194, a first compile attempt that failed with HTTP 400 $0.049047, evaluate $0.353626, smoke $0.070812, in all $0.912838, plus $0.003976 of embeddings; 8,601,840 of 9,936,337 prompt tokens were cache hits (`spend`) | `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` | +| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` | +| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record | + +The increment-2 record's note on the two denominators: "the reference divides by the 46 +candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is +0.2% under". + +!!! note "The meter prices at list; the account moved less" + The attribution table prices every call at the Model's list price + (`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution + table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the + DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75 + after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the + provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered): + DeepSeek's off-peak discount window, presumably; the meter prices at list" + (`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the + meter, never against the balance. + +### Not done yet + +- **No model-side token budget in ax itself.** The plane's requirements state what ax does + not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek + provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until + then the ceiling lives in the metered client, not in the executor. +- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded + triage and adds that recording's cost as its own line. A triage task is planned as a later + specification. +- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5} + (`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision + engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule + sets and features [are] compared by controlled experiments, so that nothing is adopted on one + run's number" (Requirement 4 of its specification). The retrieval cap of at most two + contrast examples from other families is a module constant (`learn/retrieve.py`, + `CONTRAST = 2`), a maintainer decision, not an experiment variant yet. + +## B. Development: the maintainers' own sessions + +```mermaid +flowchart LR + session["agentic coding session"] --> read["Read of a whole file"] + read --> guard{"read_guard: over 350 lines?"} + guard -- "no" --> allow["read proceeds"] + guard -- "yes" --> deny["denied: read a range or ask the bulk reader"] + deny --> bulk["bulk_read.py: cheap model answers with path:line under --max-usd"] + session --> worktree["subagent in a worktree: its tool output never enters the main context"] + session --> files["scripts in files, terse output"] + transcript["session transcript .jsonl"] --> report["token_report.py: share per content kind"] + report --> record["counts-only dev_token_report in the measurement record"] +``` + +### The measurement: tool I/O dominates + +`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an +argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and +share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`, +`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are +approximated at four characters each. It prints to stdout and writes nothing; what gets +committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json` +(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"): +7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1, +`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the +session is paid for (assistant text and thinking) is the small remainder. That is why the +remedies below target file contents, not prose. + +### The remedies in `benchmarks/dev/` + +**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on +stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines +(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file +within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line +whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The +script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked +file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command +repo-relative because hooks run from the project directory: + +```json +{ + "hooks": { + "PreToolUse": [ + { + "matcher": "Read", + "hooks": [ + {"type": "command", "command": ".venv/bin/python benchmarks/dev/read_guard.py"} + ] + } + ] + } +} +``` + +**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question, +citing `path:line` for every claim; it reports what the files say and decides nothing. + +```text +.venv/bin/python benchmarks/dev/bulk_read.py "" FILE [FILE ...] [--max-usd 0.05] +``` + +Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered +lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`, +otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is +checked before every chunk; at the ceiling the answer ends with +`[stopped at the $X ceiling before: ]`. The answer goes to stdout, the summary +`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2. + +**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described +above. + +### Practices + +- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output + `tool_result`; both are paid again at every later turn of the session. A script on disk is + read once, by the interpreter. +- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness + fails loudly with the exit code rather than printing a table that has to be read back. +- **Subagents with worktrees** for work whose output need not enter the main context: a + subagent's reads and command output stay in its own context and only its conclusion comes + back; a git worktree keeps its edits off the main checkout. +- **Counts-only measurement records.** A record under `benchmarks/measurements/` or + `benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so + it can be quoted on a page like this one and reread by a model at a few hundred tokens. + +Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for +the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are +measured against. diff --git a/mkdocs.yml b/mkdocs.yml index 43f55dc..3b6d312 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,6 +56,7 @@ nav: - Memory: memory.md - Memory and detection: memory-and-detection.md - Decision engine: decision-engine.md + - Token ergonomics: token-ergonomics.md - Evaluation: evaluation.md - Using it: - Examples: examples.md