Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

TL;DR Highlight

Local LMs answered 88.7% of real queries, and IPW rose 5.3x from 2023 to 2025.

Who Should Read

Backend developers weighing whether to move some requests to local models because of LLM API cost and privacy concerns

Core Mechanics

Local LMs handled 88.7% of queries
- The study evaluated 20+ local LMs on 1M real single-turn chat and reasoning queries.
- Accuracy was measured as win rate against frontier models and varied by domain.
They propose a new metric called IPW
- IPW (intelligence per watt) is task accuracy per unit of power.
- It compares capability and efficiency across model-accelerator pairs with one number.
IPW improved 5.3x from 2023 to 2025
- Both algorithmic and accelerator advances contributed.
- The share of locally serviceable queries rose from 23.2% to 71.3%.
Local accelerators lag cloud ones
- Running identical models, local accelerators had at least 1.4x lower IPW than cloud accelerators.
- The authors see significant headroom for optimizing local accelerators.
Eight hardware accelerators were tested

Community Reactions

Experience that latency is lower locally
- One commenter said a local model starts acting before a cloud service finishes login checks and queueing.
- They also cited predictability and avoiding personal data collection.
Pushback that it does not measure intelligence
- Some said calling these benchmarks a measure of intelligence is generous.
- Another said intelligence is not linear, so an "x per y" metric breaks down.
Skepticism about local energy efficiency
- A commenter found it hard to believe local uses less energy given cloud batching.
- Another noted the paper does not state the M4 Max RAM size.
Opinion that per-request routing hurts in the cloud
- One commenter said per-request model routing breaks KV cache reuse and raises energy and cost.
- They argued for per-session routing and cache-aware local/cloud hybrid serving.
Experience running fully local

How to Apply

Classify last month's request logs by difficulty
- Pull 100 sample requests today and split them into simple transforms, summaries, classification versus complex reasoning.
- The simple share is your candidate pool for local models.
Measure a local model's win rate on candidates
- Run a small model on ollama against the same queries and compare with your current API model's answers.
- Use win rate against the current model's answers as the criterion.
Route per session, not per request
- Pin each session to one model on the cloud path so the KV cache stays warm.
- Record latency, power, and accuracy together when comparing local and cloud costs.

Try It

copy & run

# Paste into Claude or ChatGPT (simplified version of the paper's method: judging locally serviceable queries)

You are an LLM routing analyst. Below is a sample of queries from our service.
{{list of 100 queries}}

Classify each query by this criterion.
- Is a small local model (20B or fewer active parameters) enough: yes/no
- Give the reason in one line

At the end, give the yes ratio (%) and 3 common traits of the queries marked no, in a table.

Next step: run 20 queries marked yes on a small model via ollama and compare with your current API model's answers.

Terminology

IPW (intelligence per watt)Task accuracy obtained per unit of power consumed. Like fuel economy for cars, it shows how smart the answers are for the same power.

Active parametersThe model parameters actually used in computation for a query. Think of it as the staff actually assigned to a job out of the whole company.

KV cacheA cache that stores earlier token computations for reuse. It is a notepad that keeps the model from re-reading the start of a conversation.

PrefillThe stage where the full prompt is first read and processed. It is the time spent reading the question before writing the answer.

BatchingGrouping many requests to process them together for efficiency. Like loading parcels onto one truck.