LLMs: Intelligence vs. Cost

Who Should Read

A backend developer using LLM APIs who must pick a model per task by cost versus capability.

Core Mechanics

• AA's chart hides the price gap with a log axis
  • ArtificialAnalysis (AA) plots Intelligence vs. cost with a logarithmic cost axis.
  • Differences among cheap models are visible, but the huge gap between frontier and cheap models is hard to feel.
  • The author cites models at $0.015 and $0.032 sharing a plot with one at $3.69.
• Prices changed to OpenRouter and local electricity
  • AA uses the developers' official API prices, but open-weights models can be rented much cheaper from third parties.
  • The author repriced models like Kimi K3 and GLM-5.3 to OpenRouter prices.
  • Models under 35B parameters are priced by local electricity instead of datacenter pricing.
• GLM-5.3 is 7.5x cheaper than Fable 5.1
  • Fable 5.1 scores 66 at $3.69 per task.
  • GLM-5.3 scores 60 at $0.49.
  • The author thinks most people can't tell the difference.
• GLM-5.3-Flash costs 1/160 of Fable
  • GLM-5.3-Flash at high effort scores 55 at $0.023 per task.

Community Reactions

• Speed should be a metric too
  • Commenters said 1-2 minutes versus 5-10 minutes per answer matters a lot.
  • One commenter runs expensive models at light reasoning to get quick, good answers.
  • Several wanted a 3D intelligence/price/speed plot, and one built a 3D app.
• Pushback that log scale is better
  • One commenter argued that switching among cheap models can double spend, which log scale reflects.
  • They said the Pareto frontier is easier to see on the log plot.
  • They also criticized zero hardware cost and the lack of explanation of how the frontier changed.
• Subscriptions change the math
  • One commenter said OpenAI and Anthropic subscriptions give about a 10x discount versus per-token prices, so those models should shift 10x left.
  • Another said hardware cost should be spread over time.
• Views on open-weights pricing
  • One commenter said that filtering by quantization makes most providers' prices similar.

How to Apply

• Sort last month's calls by task difficulty
  • Pull simple calls like summarization, classification, and extraction from today's logs.
  • Run the same inputs through a cheap model such as GLM-5.3-Flash and compare.
  • Move the task to the cheap model if quality holds.
• Re-compare open-weights models at OpenRouter prices
  • Build a table of per-provider OpenRouter prices instead of official API prices.
  • Exclude providers that are slow or lack Zero Data Retention.
  • Match quantization levels when comparing.
• Split model routing into two tiers by difficulty
  • Send only hard tasks like one-shotting a whole project to the top model.
  • Default everything else to a mid or cheap model.
  • Log cost and latency per task and adjust the threshold.
• Separate electricity from hardware in local cost
  • Count only electricity if you already own the machine.

Try It

# Paste into Claude or ChatGPT as-is (a simplified prompt version of the article's idea)

I use LLM APIs for the tasks below. Classify each task into three tiers: "needs top model / mid model is enough / cheap or local model is enough".

Task list:
{{tasks: name, monthly calls, avg input/output tokens}}

Models I use now and per-token prices:
{{models and prices}}

For each task, give a table with:
1. The tier and why
2. Where quality might drop if I downgrade the model
3. A small test to verify that risk

Next step: pick 10 tasks I plan to downgrade and ask for a side-by-side comparison of the two models' outputs.

Terminology

Related Papers

• Mercury 3: Diffusion at scale

  Scaling diffusion LLM 40x gives Mercury 3: 4–8x faster than speed-tier models, similar quality

• Timestral: Mistral's time-series foundation models

  A 209M time-series model ranks first on three benchmarks and runs on CPU.

• DeepSeek V4.1 Flash 1017 (Max)

  DeepSeek V4.1 Flash 1017 Max scores 45 on the Intelligence Index at 211 tok/s and $0.30 input.

• If you don't run your own LLM proxy, how do you know whats going on?

  Running your own LLM proxy shows which keys, models, and cache misses drive your usage.

• DeepSeek: DeepSeek v4.1 Flash 1017

  DeepSeek V4.1 Flash 1017: $0.05/M input, 1M context, built for agents

• DeepSeek-v4.1-Flash-1017

  DeepSeek V4.1 Flash minor update lifts coding and agent benchmarks while cutting run time and output length

• Show HN: Llmspeed – feel how fast a local LLM runs before you buy the hardware

Related Resources

• https://openteams.com/intelligence-vs-cost/
• https://artificialanalysis.ai/#intelligence-comparison-tabs
• https://www.williamangel.net/apps/model_performance.html