Token Reduction Is Not Cost Reduction | AI Paper Digest

TL;DR Highlight

Cutting Claude Code tool output by 38.4% raised billed cost by 6.8%

Who Should Read

A developer adding token-compression tools to a coding agent like Claude Code to cut API spend

Core Mechanics

38.4% less tool output, +6.8% cost
- RTK-ML cut tool-output tokens from 5.62M to 3.46M.
- Its billed cost still rose 6.8% over baseline.
Cache traffic dominates the bill
- Cache creation plus cache reads were about 80% of the actual bill.
- Generated output tokens were only 10.4% of the bill.
Only about 6% of cost is reachable
- The harness base (system prompt and tool definitions) and hidden thinking carry 94.0% of cost.
- Tool outputs are 3.3% of cost, so the ceiling on compression savings is small.
Compression makes the agent redo work
- Missing context led to extra retrieval, diagnosis, and testing turns.
- Each added turn re-sends the whole context prefix and cancels the savings.
Compression broke edit anchors and pipes
- SEARCH/REPLACE edits need byte-exact code anchors, and compression altered them.

Evidence

RTK-ML: +6.8% billed cost
- Versus baseline Claude Code, +6.8% [+2.8, +11.3] over 103 tasks and 712 runs per arm.
- Cost per successful execution (CPS) ratio was 1.051.
Headroom v0.27.0: +48.4% billed cost
- Versus baseline, +48.4% [+42.3, +55.0] with a CPS ratio of 1.464.
- Success difference was +1.4pp [-0.2, +3.1], not significant.
Tool-output reduction barely predicts cost
- Across 100 Haiku tasks, Pearson r=0.154 [-0.051, +0.356] and Spearman ρ=0.013.
Patch application fell in the Go editing study
- On 40 ContextBench-Go rows, compression cut billable tokens by 73% and cost by 59%.
- Patch applies dropped from 27/40 (raw) to 15/40 (compressed).

How to Apply

Measure cost per success, not tokens saved
- Pick one compression tool you use today and run the same 10 tasks with it on and off.
- Divide total billed cost by the number of successful runs and compare.
Log cache read and write tokens separately
- Record input_tokens, cache_creation_input_tokens, cache_read_input_tokens, and output_tokens for every run.
- Check which component dominates the bill before choosing what to compress.
Add exclusion rules to compression
- Pass tracebacks, test output, and patch anchors through unchanged.
- Never compress stdout that is piped into another program.
Compare turns and success alongside cost
- Record average turns, re-search counts, and success rate before and after adopting a tool.
- If turns and cost went up, turn the tool off or narrow its scope.

Try It

copy & run

# Paste into Claude Code or Claude/ChatGPT (simplified version of the paper's method: cost per success)

Below is a log of coding-agent runs. Each run has arm (baseline/compressed), cost_usd, success (true/false), and turns.
{{run log CSV}}

Compute and show in a table:
1. Per arm: number of runs, success rate, average turns, total cost
2. Per arm: cost per success = total cost / number of successes (failed runs stay in total cost)
3. Compressed vs baseline: cost change (%) and cost-per-success ratio

Give a one-line verdict on whether compression pays off by cost per success, not by token reduction.

Next to try: pair runs by task and list only the tasks where turns increased.

Terminology

prompt cacheA feature where the server stores the front part of a prompt and re-reads it cheaply on later requests. Think of it as keeping a copy of the document's opening for reuse.

CPS (cost per successful execution)Billed cost per successful run. It is an arm's total billed cost divided by its number of successes, with failed runs' cost included.

trajectoryThe sequence of turns an agent takes to finish a task. The longer it gets, the more often the context is re-sent.

SEARCH/REPLACE editingAn edit format where the model writes the exact code block to find and its replacement. If the anchor differs by even one character, the patch fails to apply.

RTK / HeadroomRTK compresses shell command output through a hook. Headroom is a proxy in front of the API endpoint that rewrites request payloads.

Related Papers

Related Resources

https://github.com/PointFiveLabs/ai-efficiency-benchmark

Original Abstract (Expand)

Token-reduction tools for coding agents are often evaluated by the number of tokens they remove, but token count alone does not determine end-to-end inference cost. We evaluate three token-reduction approaches against an unmodified Claude Code baseline across controlled coding tasks, measuring provider-billed cost, task success, cache traffic, and agent behavior. The largest compression setup reduced delivered tool-output tokens by 38.4% but increased billed cost by 6.8%, while lighter compression produced only small and statistically uncertain savings. Across tasks, token reduction was weakly correlated with cost reduction (Pearson r = 0.15). Cost decomposition shows that prompt-cache creation and reads dominate the measured input-side cost, leaving only a limited fraction of total spend directly addressable by tool-output compression. We also find that compression can alter agent trajectories through additional retrieval, diagnosis, testing, and turns, offsetting local token savings. On a SWE-bench Go subset, aggressive compression also reduced successful patch application. These results show that token reduction is not a reliable proxy for cost reduction in tool-heavy coding agents. Effective optimization should therefore be evaluated at the level of cost per successful task, including cache behavior, trajectory changes, and correctness rather than token counts alone.