Whose turn is it? Context-guided model routing for stateful agents | AI Paper Digest

TL;DR Highlight

A router that reads session history passed 3.3 points more tasks than one without it.

Who Should Read

A backend developer mixing cheap and expensive models in a multi-turn agent to cut LLM costs.

Core Mechanics

Giving the router history raised pass rate
- At a matched offloading rate, the router with history passed 28.3% versus 25.0% without.
- The gain came from which turns were offloaded, not how many.
Follow-ups like "apply the same format to Q3" can't be judged without history
- Without prior turns, follow-up requests look deceptively simple or ambiguous, so the router misassigns them.
- The paper therefore frames routing as a sequential decision problem.
Cheap per token is not cheap per task
- Haiku 4.5 has one-fifth of Sol's input token price but made about 41 calls and used 1.6M input tokens.
- Its per-task cost advantage was only 3.2x.
Longer sessions shrink the cheap model's discount
- Every call re-reads the accumulated session, so extra calls cost more.
- Mean context per call roughly doubled, from about 18K to about 38K tokens.
Switch cost depends on direction because of caching

Evidence

History ablation on SheetCopilot
- Router with history scored 28.3% versus 25.0% without, a 3.3-point gain.
- Cheap-model share was nearly equal: 22.7% versus 23.3%.
- Gains per pair were 5.4 (Sol/Haiku), 2.7 (Sol/Luna) and 1.7 points (Opus high/low).
History router kept expensive-model quality
- On SheetCopilot: expensive-only 27.7%, cheap-only 26.8%, history router 28.3%.
- On SpreadsheetBench 2: 29.3%, equal to expensive-only, versus 26.6% for cheap-only.
Random routing and de-escalation did poorly
- Random routing averaged 22.2% on SheetCopilot and 26.6% on SSB-2.
- De-escalation cost $2.440 on Opus high/low, above both fixed models.
Cache analysis and cost ratios
- Downward-switch break-even was k*=1.25 calls for Sol→Luna and Sol→Haiku.
- An upward switch cost about $4.9 per million context tokens, about 10x the $0.5 of a downward switch.
- Haiku's per-task cost advantage shrank from 6.8x on SheetCopilot to 3.2x on SSB-2.

How to Apply

Add history to your router prompt
- If your router sees only the current request, include recent requests, tool results and errors.
- Compare routing decisions with and without history on the same traffic logs.
Choose the cheap model by realized cost per task
- Run candidate models on your real workload and record calls, input tokens and cost per task.
- Don't decide from the token price list alone.
Switch down early, avoid switching up late
- Once early turns look easy, drop to the cheap model and stay there.
- Allow upward switches in long contexts only when the quality gain is large.
Compare the candidate pair first
- If the cheap model alone passes about as often as the expensive one, skip the router and use it.
- Adopt routing only for pairs with a large quality gap.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: history-aware turn router)

You are an agent's model router. Given the session history and the new request, decide whether this turn goes to the cheap model (CHEAP) or the expensive model (EXPENSIVE).

[Session history]
{{earlier requests, agent responses, tool call results, errors}}

[New request]
{{current user request}}

Criteria:
- Planning, coordinating across artifacts, or recovering from an earlier error -> EXPENSIVE
- Simple extension of results already built in earlier steps -> CHEAP
- If the request is ambiguous without history, decide based on the history

Output one line of JSON: {"route": "CHEAP" or "EXPENSIVE", "reason": "one sentence"}

Next, try this: rerun the same request with the history section left empty and compare how the decision changes.

Terminology

Model routingA dispatcher that picks which LLM handles each request. It's like assigning easy tasks to a junior and hard ones to a senior.

Offloading rateThe share of turns sent to the cheap model. It shows how much senior work was handed to the junior.

Prompt cacheA feature that bills repeated prompt prefixes at a lower rate. It is stored per model, so switching models resets it.

Escalation / De-escalationEscalation starts on the cheap model and hands off to the expensive one when needed. De-escalation does the opposite.

Zero-shot routerA router that decides using only a prompt and rules, with no routing labels or fine-tuning.

Related Papers

Original Abstract (Expand)

Model routing can reduce the cost of LLM applications by assigning simpler requests to cheaper models without sacrificing the quality of the response. However, most routers treat requests as independent. Stateful agents do not operate under this assumption: a user turn may require multiple model and tool calls, its difficulty depends on the state produced by earlier turns, and changing models can sacrifice reuse of the conversational prompt cache. We study zero-shot, cost-aware routing at the user-turn boundary, systematically varying the routing rule, access to session history, candidate models that differ in size, reasoning effort, or model provider, and prompt-cache behavior. We evaluate on multi-turn versions of SheetCopilot and SpreadsheetBench 2, where persistent workbooks and cross-turn dependencies make isolated routing decisions especially brittle. At a matched offloading rate, giving the router session history improves task pass rate over the same router without history, with gains for every model pair; history-aware routing remains within one point of always using the expensive model. We also found that cost depends on execution dynamics rather than token prices alone: the cheaper model's per-task cost advantage narrows from 6.8x to 3.2x on longer sessions. A closed form cache analysis further shows that downward switches (expensive to cheap) recover their cache cost within two LLM calls, whereas upward switches do not break even under the evaluated conditions. Together, these results show that agent routing is a sequential decision problem: effective policies must account for what happened earlier in the session and for the asymmetric cost of switching models.