Everyone is building LLM routers, we deprecated ours

TL;DR Highlight

After four months, per-request LLM routing cost more than it saved, so Manifest shut it down.

Who Should Read

A backend or agent developer weighing an LLM router that splits requests across models to cut costs.

Core Mechanics

Manifest shut its router down
- Launched in March, deprecated in June, and shutting down for good on September 1st.
- It sorted requests into four tiers (simple, standard, complex, reasoning) to cut costs.
- After four months with 7000 cloud users, results were mixed and GitHub issues and discussions piled up.
The prompt alone doesn't reveal difficulty
- The prompt is just the trigger, not the whole task.
- Context that determines complexity shows up later through tool calls and web searches.
- 'Evaluate and improve the tests' is trivial for a plain HTML5 site but huge for the Linux kernel repo.
Caching beats routing on cost
- Cache reads are 75% to 90% cheaper than uncached inputs.
- System prompts and conversation history sit at the start of the prompt, so prefix cache works well.
- A cache-aware router ends up sticking to the first model, so it effectively doesn't route.
Switching models mid-session breaks consistency

Community Reactions

Difficulty can't be known upfront, per one commenter
- Someone who researched routing last year reached the same conclusion.
- Difficulty depends on what the agent can retrieve, so the query alone doesn't tell you.
Pinned subagent roles worked well for one commenter
- An orchestrator builds context and assigns models to fixed subagent roles.
- Using Minimax M3 for exploration and librarian tasks was fast and cheap, and a $10 plan lasts the whole month.
Small pools fixed caching, per one commenter
- Keep the model pool small with clearly differentiated models.
- They reported over 99% cache hits routing between GPT 5.4 and DeepSeek.
Counterpoint: context-aware routers work
- One commenter got good results with routers that understand their workloads.
- Another said difficulty buckets fail, but routing chatbot prompts by category like legal, medical, and code works well.

How to Apply

Check your prompt cache hit rate
- Today, log the cached input token share from the usage field of your API responses.
- Keep the system prompt and history at the front of the prompt to preserve prefix matches.
Pin models by role instead of routing
- Write the model and effort for each agent step (explore, summarize, code) in a config file.
- Decide by call site, not by classifying request content.
Calculate savings before adopting a router
- Compare current cost with the 75-90% cache discount applied against estimated routing cost.
- Include the labor of maintaining evals, prompts, and monitoring per model.

Try It

copy & run

# Paste into Claude or ChatGPT as is. A checklist prompt to apply the article's claims to your service

Our service's LLM call structure is below.
{{per-call description: which model is used at which step, system prompt length, conversation length}}

Review it from these angles.
1. Assuming cache reads are 75-90% cheaper than inputs, how should I restructure prompts to use prefix cache better?
2. Compare adding a content-based model router with pinning models by call step (role). Use cost, consistency, and eval/monitoring burden.
3. Find cases in our service where difficulty can't be known from the prompt alone.

Next step: ask for a table of the pinned-model plan per step.

Terminology

LLM routerA traffic cop that picks which model answers each request. Simple ones go to cheap models and hard ones to expensive ones.

prefix cacheReuses already computed results when the start of a prompt is identical. Like resuming a book from a bookmark instead of rereading the intro.

stickinessKeeping requests on the model chosen first. It preserves cache but weakens the reason to have a router.

effort parameterA setting that controls how deeply the model thinks. The same model gives different speed and quality at different values.