docs: token ergonomics -- runtime routing/budgets/caches and the development-side tooling

One page, two halves with a Mermaid diagram each. A: the plane's token economy (one Model per Task with a metered usd/calls budget, attribution table, credentials only in the start request, repo-facts and content-addressed reuse, replay at $0, triage-hunt-tie-break order, prefix-cache prompt order) with every cost quoted from plane-increment-1/2.json, the injection-slice and harvest records, and an honest not-yet list (no token budget in ax, triage applied not run, k and contrast fixed outside experiments). B: the development-side measurement (tool I/O 85.5%, previously 90%) and the benchmarks/dev remedies: read_guard hook, bulk_read portal, token_report, plus the practices. Linked from the nav, index, memory-and-detection and the README docs list. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
2026-10-16 03:49:26 +00:00 · 2026-10-02 10:42:17 +02:00
parent 39b0c0a9d1
commit 6d601c1464
5 changed files with 241 additions and 1 deletions
@@ -229,6 +229,8 @@ Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra
  pair corpora and the plane increment results.
 - [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine.
 - [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening.
+- [docs/token-ergonomics.md](docs/token-ergonomics.md): model spend at runtime (budgets, reuse,
+  measured costs) and in coding sessions.
 - [docs/examples.md](docs/examples.md): end-to-end walkthroughs.

 ## Development
@@ -49,6 +49,8 @@ flowchart LR
 - Understand a finding's evidence and score: [Architecture](architecture.md).
 - Operate the plane and its store: [ax on this host](ops/ax/README.md) and
  [RustFS setup](rustfs.md).
+- Where the model spend goes, per Task at runtime and in the maintainers' own coding sessions,
+  and what is reused instead of paid again: [Token ergonomics](token-ergonomics.md).
 - Trust boundaries and hardening: [Threat model](threat-model.md).

 Every number in these pages is quoted from a committed record whose path is given next to it.
@@ -141,7 +141,8 @@ Record: `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/reco
 only). Family **injection**; memory of 724 static examples from all families; input profile v1;
 framework priors off; out-of-repository evaluation (compile split never evaluated, outer grouped
 5-fold, calibration cross-fitted, operating points nested); 95% repository-cluster bootstrap
-intervals.
+intervals. The slice's spend ($0.003215 per candidate at evaluation, k = 5, and its replay at
+$0) is on [Token ergonomics](token-ergonomics.md#measured-costs).

 | Measure | Value | What it means |
 | --- | --- | --- |
@@ -0,0 +1,234 @@
+# Token ergonomics
+
+Model tokens are the one running cost of this project, and they are spent in two places: by the
+**plane** when a Run verifies candidates or compiles the decision engine, and by the maintainers'
+own **agentic coding sessions**. The maintainer's framing (2026-09-29): "the token saving strategy
+is a routing concern. ax can handle that and structure our services in small components." So the
+runtime half is routing, one Model per Task with its own budget, and reuse of what was already
+paid for; the development half is tooling that keeps bulk file contents out of the session.
+
+Every number on this page comes from a committed record, named next to it. Records are counts
+only; none holds code, prompts or responses.
+
+## A. Runtime: the plane's token economy
+
+```mermaid
+flowchart LR
+    subgraph budget ["Budget and attribution"]
+        start["start request (credentials only here)"] --> task["Task: one Model, budget usd and calls"]
+        task --> meter["metered client refuses the next call at the ceiling"]
+        meter --> summary["summary.json: calls, tokens, usd"]
+        summary --> status["ousast plane status: attribution.json per task and per Model"]
+    end
+    subgraph reuse ["Pay once, reuse"]
+        facts["repo-facts, model-free"] -- "same repo, pin, candidates, image" --> seeded["seeded: calls 0, no usd"]
+        blobs["excerpts, embeddings, responses, programs by sha256"] -- "replay" --> zero["rerun at $0"]
+    end
+    subgraph order ["Cheap before expensive"]
+        triage["recorded triage per file"] --> hunt["one hunt per file, up to 6 candidates"]
+        hunt --> agree["agree: a and b"]
+        agree -- "disputed only" --> tiebreak["pass c, 2-of-3"]
+    end
+```
+
+### One Model per Task, one budget per Task
+
+A Run is a DAG of ax Tasks; each Task binds at most one Model and its own `budget: {usd, calls}`
+([The plane on ax](plane.md)). The budget is enforced by the metered client
+(`src/openultrasast/plane/budget.py`, `MeteredClient`): `check()` runs before every call and
+raises `BudgetExhausted` once the spent `usd` or the call count reaches the ceiling, so the call
+that crosses the ceiling completes and the next one is refused. The task then ends `unfinished`
+and resumes from its `units.jsonl` on a rerun with a larger budget; an HTTP 401/402 or
+"Insufficient Balance" is an `AccountError`, which fails the task and starts nothing further.
+Usage is attributed per call: the wrapped client appends one usage row per call
+(`prompt_tokens`, `prompt_cache_hit_tokens`, `completion_tokens`), rows are absorbed even when
+the call raises, and they are priced by the Model's price list. An unpriced Model cannot run
+under a `usd` budget, and `summary()` reports `usd: None` for it, never 0. Model-free tasks run
+with `{usd: 0, calls: 0}`, so a stray call fails loudly.
+
+`ousast plane status RUN` is the attribution table: one row per task with `calls`, `prompt`,
+`cache_hit`, `output` and `usd` from each task's `summary.json`, a `subtotal` row per Model and a
+`total` (a missing value makes a sum `n/a`, never a smaller number); `--units` adds the per-unit
+rows. The same table is written as `attribution.json` in the run directory
+(`src/openultrasast/plane/reconciler.py`, `attribution()`/`status()`). Provider credentials
+reach a task only in the start request: `router.py` reads the variable the Model's
+`secretKey.key` names, the runner keeps the value in memory, passes it to the command's
+environment and redacts it from echoed stderr; it never appears in a manifest, a rendered Task
+or a file.
+
+### Pay once, reuse
+
+- **Repo facts are computed model-free and reused.** `repo-facts` reads source text only (no
+  model, no git) and its `facts.json` is byte-identical across runs over the same tree. The
+  memory store keeps a facts entry under the key `(repo, pin, candidates digest, runner image
+  digest)` (`src/openultrasast/plane/memory.py`, `memory_key()`); when a later Run's task
+  carries the same key, `seed()` writes the stored `facts.json`, a `done` summary with
+  `calls: 0` and no `usd`, and marks the task done without starting an actor. A changed pin,
+  candidate set or image finds nothing and recomputes.
+- **Content-addressed memory** (`src/openultrasast/learn/`). Excerpts are stored as
+  `excerpts/<sha256 of the normalised text>.txt`; an embedding lives at
+  `embeddings/<model>/<excerpt sha>.json` and "an excerpt is embedded once; a rerun makes no
+  call" (`embeddings.py`). Model responses are cached under the sha256 of
+  `{model, messages, params, temperature, sample}` (`program.py`, `request_key()`), and a
+  compiled program is `programs/<sha256 of its artifact>.json` with the digests of the memory
+  snapshot and of the response cache it was built from (`compile.py`).
+- **Replay is free.** With no client, the response cache replays at $0 and a miss raises
+  `ReplayMiss` instead of calling. The injection-slice record states it: "evaluate re-run
+  replay-only on the committed code: 550 responses replayed, $0, identical overall and
+  calibration" (`benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json`,
+  `replay_check`).
+
+### Cheap before expensive
+
+- **Triage first.** Verification runs only over the candidates the recorded triage kept; the
+  plane applies that recording without a model call and prices it as a separate line
+  (`recorded_triage_usd`), so every per-candidate cost below is reported both with and without
+  it (`src/openultrasast/plane/tasks/agree.py`).
+- **Hunts are batched per file.** `verify` runs one tool hunt per file for up to `PER_HUNT = 6`
+  candidates, with the candidates' known callers (capped at 8) in the prompt
+  (`src/openultrasast/plane/tasks/verify.py`, `units_of()`).
+- **The tie-break pass runs only on disputes.** `agree` compares passes a and b; pass c is
+  started with `inputs.only` set to the case's `disputed.json`, an empty list ends it `done`
+  with no model call, and the final verdict is 2-of-3 ("pass c alone never decides",
+  `agree.py`).
+- **Cache-friendly prompt order.** The decision engine's prompt is the compiled prefix
+  (instruction and fixed demonstrations, identical for every candidate so the provider's
+  prefix cache hits), then the retrieved examples, then the candidate (`learn/program.py`).
+  Its self-consistency `k` is tied to that cache: "k = 5 when samples 2..k hit the prefix cache
+  for >= 0.8 of their prompt tokens"; the measured repeat-hit share was 0.96 (injection-slice
+  record, `smoke.k_rule`, `smoke.k5_repeat_hit_share_measured`). The verify hunts share a prefix
+  too: in the first plane increment 2,385,152 of 3,820,395 prompt tokens were cache hits
+  (`benchmarks/independent/plane-increment-1.json`, `detail.cache_hit_tokens`,
+  `detail.prompt_tokens`; its reading: "Cost fell 17% (62% of prompt tokens were cache hits)").
+
+### Measured costs
+
+| Measurement | Spend | Record |
+| --- | --- | --- |
+| Plane increment 2: verify a, b and the tie-break on the 46-candidate validation set | $0.0205 per candidate over the 46 including the recorded triage, against the $0.022 reference (`gate.cost_per_candidate_with_recorded_triage_usd`); totals: plane $0.8567, recorded triage $0.0872, with triage $0.9439, reference $1.0123 (`detail.usd`); the tie-break asked 10 disputed candidates and flipped 5 to agreed | `benchmarks/independent/plane-increment-2.json` |
+| Plane increment 1: verify a and b alone | 357 calls, 54 hunts at 5.37 turns per hunt, $0.7471 plus $0.0872 recorded triage (`detail`) | `benchmarks/independent/plane-increment-1.json` |
+| Decision engine, injection slice: evaluation at k = 5 | $0.003215 per candidate (`spend.cost_per_candidate_evaluate_usd`); metered at list price: canary $0.010158, compile $0.429194, a first compile attempt that failed with HTTP 400 $0.049047, evaluate $0.353626, smoke $0.070812, in all $0.912838, plus $0.003976 of embeddings; 8,601,840 of 9,936,337 prompt tokens were cache hits (`spend`) | `benchmarks/measurements/2026-10-01-decision-engine-injection-slice/record.json` |
+| Decision engine harvest: `verify` part | $2.5373 over 1,961 calls, 474 tasks done, 10,921,167 prompt tokens of which 6,275,060 cache hits, 306,941 output tokens, task budgets summing to $8.624 under a $10 ceiling (`verify`) | `benchmarks/measurements/2026-10-01-decision-engine-harvest/record.json` |
+| Decision engine harvest: `roles` part | $1.4036 over 1,034 calls, 151 tasks done, 2,398,669 prompt tokens of which 329,600 cache hits, 370,155 output tokens, task budgets summing to $4.917 under a $10 ceiling (`roles`) | same record |
+
+The increment-2 record's note on the two denominators: "the reference divides by the 46
+candidates before triage; on that basis the plane is 7% cheaper, on the 43 after triage it is
+0.2% under".
+
+!!! note "The meter prices at list; the account moved less"
+    The attribution table prices every call at the Model's list price
+    (`plane/models/deepseek-flash.yaml`). The harvest record: "Spend is the plane's attribution
+    table (`ousast plane status`, task meters priced at plane/models/deepseek-flash.yaml); the
+    DeepSeek account moved less (balance below)", with the balance at $19.63 before, $18.75
+    after `verify` and $18.23 after `roles`. The injection-slice record names the ratio: "the
+    provider charged ~1/3 of the list-price meter (balance moved $0.30 for $0.91 metered):
+    DeepSeek's off-peak discount window, presumably; the meter prices at list"
+    (`spend.balance_note`, balance $18.2 to $17.9). Budgets and gates are set against the
+    meter, never against the balance.
+
+### Not done yet
+
+- **No model-side token budget in ax itself.** The plane's requirements state what ax does
+  not yet carry: "task dependencies, artifacts, token budgets (on ax's roadmap), a DeepSeek
+  provider"; the thin `Run` layer carries them and dissolves into ax as ax gains them. Until
+  then the ceiling lives in the metered client, not in the executor.
+- **Triage is applied, not run.** The plane has no triage task; it filters by the recorded
+  triage and adds that recording's cost as its own line. A triage task is planned as a later
+  specification.
+- **`k` and the contrast examples are not knobs.** `k` is a compiled setting in {1, 3, 5}
+  (`learn/compile.toml`, "k and lam remain A/B variants"), changed only through the decision
+  engine's A/B experiments, its requirement that "changes to prompts, models, pass counts, rule
+  sets and features [are] compared by controlled experiments, so that nothing is adopted on one
+  run's number" (Requirement 4 of its specification). The retrieval cap of at most two
+  contrast examples from other families is a module constant (`learn/retrieve.py`,
+  `CONTRAST = 2`), a maintainer decision, not an experiment variant yet.
+
+## B. Development: the maintainers' own sessions
+
+```mermaid
+flowchart LR
+    session["agentic coding session"] --> read["Read of a whole file"]
+    read --> guard{"read_guard: over 350 lines?"}
+    guard -- "no" --> allow["read proceeds"]
+    guard -- "yes" --> deny["denied: read a range or ask the bulk reader"]
+    deny --> bulk["bulk_read.py: cheap model answers with path:line under --max-usd"]
+    session --> worktree["subagent in a worktree: its tool output never enters the main context"]
+    session --> files["scripts in files, terse output"]
+    transcript["session transcript .jsonl"] --> report["token_report.py: share per content kind"]
+    report --> record["counts-only dev_token_report in the measurement record"]
+```
+
+### The measurement: tool I/O dominates
+
+`benchmarks/dev/token_report.py` reads a Claude Code session transcript (`.jsonl`; without an
+argument, the newest under `~/.claude/projects/`) and reports, per content kind, characters and
+share: `tool_use_input` (the tool call's input as JSON), `tool_result`, `thinking`,
+`assistant:text`, `user:text`, plus the tool-call count and the top five tools. Tokens are
+approximated at four characters each. It prints to stdout and writes nothing; what gets
+committed is the counts-only summary. The one in `benchmarks/independent/plane-increment-1.json`
+(`dev_token_report`, for "the whole development session (c09e30b2), not the increment alone"):
+7.3M unique characters (~1.81M tokens), `tool_use_input_pct` 45.4, `tool_result_pct` 40.1,
+`tool_io_pct` 85.5 against `previous_tool_io_pct` 90, over 2,701 tool calls. The reasoning the
+session is paid for (assistant text and thinking) is the small remainder. That is why the
+remedies below target file contents, not prose.
+
+### The remedies in `benchmarks/dev/`
+
+**`read_guard.py`**, a `PreToolUse` hook for agentic coding sessions. It reads the hook event on
+stdin and, for the `Read` tool only, denies a whole-file read of a file longer than 350 lines
+(`LIMIT = 350`); a read with `offset` or `limit`, a missing path, an unreadable file or a file
+within the limit passes. It always exits 0 ("never fails the tool"). A denial is one JSON line
+whose `permissionDecisionReason` tells the agent to read a range or to ask the bulk reader. The
+script's docstring records that it is wired in `.claude/settings.local.json` (a local, untracked
+file) as `PreToolUse` with matcher `Read`; the setting has this shape, with the command
+repo-relative because hooks run from the project directory:
+
+```json
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Read",
+        "hooks": [
+          {"type": "command", "command": ".venv/bin/python benchmarks/dev/read_guard.py"}
+        ]
+      }
+    ]
+  }
+}
+```
+
+**`bulk_read.py`**, the portal pattern: a cheap model reads the files and answers one question,
+citing `path:line` for every claim; it reports what the files say and decides nothing.
+
+```text
+.venv/bin/python benchmarks/dev/bulk_read.py "<question>" FILE [FILE ...] [--max-usd 0.05]
+```
+
+Files go to the model in 400-line chunks with a `--- path (lines a-b)` header and numbered
+lines; the client is the project's detector client (`deepseek-flash` through `DEEPSEEK_API_KEY`,
+otherwise OpenRouter), each call has a 90 s timeout and no tools. `--max-usd` (default 0.05) is
+checked before every chunk; at the ceiling the answer ends with
+`[stopped at the $X ceiling before: <chunk header>]`. The answer goes to stdout, the summary
+`[bulk_read: N chunk(s), $X.XXXX]` to stderr. Without a chat client it exits 2.
+
+**`token_report.py`**: `python benchmarks/dev/token_report.py [TRANSCRIPT.jsonl]`, described
+above.
+
+### Practices
+
+- **Scripts into files, not heredocs.** A heredoc is `tool_use_input` and its output
+  `tool_result`; both are paid again at every later turn of the session. A script on disk is
+  read once, by the interpreter.
+- **Terse outputs.** Commands print counts, exit codes and the lines that matter; a harness
+  fails loudly with the exit code rather than printing a table that has to be read back.
+- **Subagents with worktrees** for work whose output need not enter the main context: a
+  subagent's reads and command output stay in its own context and only its conclusion comes
+  back; a git worktree keeps its edits off the main checkout.
+- **Counts-only measurement records.** A record under `benchmarks/measurements/` or
+  `benchmarks/independent/` holds numbers and their keys, never code, prompts or responses, so
+  it can be quoted on a page like this one and reread by a model at a few hundred tokens.
+
+Related: [The plane on ax](plane.md) for the Run and Task contract, [Memory](memory.md) for
+the store the reuse keys live in, [Evaluation](evaluation.md) for the gates these costs are
+measured against.
@@ -56,6 +56,7 @@ nav:
      - Memory: memory.md
      - Memory and detection: memory-and-detection.md
      - Decision engine: decision-engine.md
+      - Token ergonomics: token-ergonomics.md
  - Evaluation: evaluation.md
  - Using it:
      - Examples: examples.md