The efficient frontier of LLM inference

TL;DR Highlight

LLM inference techniques either slide along the latency-throughput curve or push the whole curve outward

Who Should Read

A backend or MLOps developer self-hosting LLMs who is unsure which serving settings to tune first to balance latency and cost

Core Mechanics

Inference techniques come in two kinds
- Moving along the curve: trade latency against throughput (which drives cost).
- Pushing the curve out: create more overall efficiency you can allocate.
Batch size is the simplest dial
- Small batches give great per-user latency but few tokens per GPU, so cost per token is high.
- Larger batches worsen per-user latency but improve throughput and lower cost.
Parallelism strategy shapes the tradeoff
- For latency, increase Tensor Parallelism (TP), since its communication is fast over NVLink.
- Lower Expert Parallelism (EP) favors latency, while wide EP across a rack favors throughput.
- Attention Data Parallelism (ADP) replicates attention layers, trading per-request speed for throughput.
Quantization pushes the curve out but trades quality
- Lower precision in weights, activations, or KV cache improves both latency and throughput.
- With microscaling formats like MXFP4 and NVFP4, big efficiency gains are possible with little-to-no quality loss.

Community Reactions

Experience running cheap self-hosted hardware showed the same techniques matter
- A commenter listed pipeline parallelism, RAM caching, speculative decoding with domain n-grams, Q5-Q6 support, FP8 contexts, paged attention, and prefix caching.
- They reported llama.cpp decode dropping to about 5 tok/s during long prefills, and vLLM/SGLang being buggy on non-standard setups.
Opinion that P/D disaggregation is the next big thing
- A commenter noted prefill is compute-bound and decode is memory-bound, so different node types could suit each.
- Another said spec-dec matured in 2026 and is adopted by all major OSS engines.
Pushback on terminology and framing
- Commenters said this is a Pareto frontier, while 'efficient frontier' is a portfolio theory term.
- Quality is a third dimension not discussed deeply, and a jagged curve needs bespoke benchmarks.
- One called the opening sentence a tautology.
Opinion that these techniques haven't changed in years
- A commenter said precision got lower and acceptance rates better, but no new concepts appeared.

How to Apply

Sort your current serving config into two buckets
- List your vLLM/SGLang settings as 'moves along the curve' or 'pushes the curve out'.
- Paste the config and traffic profile into the prompt below to get the classification.
Sweep batch size to draw your own curve
- Run several batch sizes and record per-user tok/s and total GPU tok/s together.
- Find the sharp bend and pick the point that meets your SLA.
Compare quantization candidates with a quality benchmark
- Enable a low-precision format like NVFP4 and measure quality drop on your own eval set.
- If the loss is acceptable, keep the latency and throughput gains.
Try speculative decoding for code-generation traffic
- Enable EAGLE-3 or similar on predictable-output workloads and compare tok/s.
- Also check whether max batch size shrinks.

Try It

copy & run

# Paste into Claude or ChatGPT (simple prompt version of the article's classification)

I want to optimize LLM inference serving. I use this classification.
- Moves along the curve (tradeoffs): batch size, parallelism strategy (TP/EP/ADP), quantization (trades with quality)
- Pushes the curve out: kernel/runtime optimization, speculative decoding, P/D disaggregation

My setup:
- Model: {{model name}}
- GPU: {{GPU type and count}}
- Serving engine: {{vLLM/SGLang etc.}}
- Traffic profile: {{e.g. coding agent, input length, output length, cache hit rate}}
- Current config: {{batch size, parallelism, precision, etc.}}
- Goal: {{latency first / cost first}}

Request:
1. Sort my current config into the two categories as a table.
2. Rank the techniques I haven't used yet by expected impact, with reasons.
3. Design a sweep experiment for each (variables to change, metrics to measure).

Next step: paste your sweep results and ask it to interpret where the curve bends.

Terminology

efficient frontierThe curve of best achievable combinations when trading two goals, like latency and throughput. Settings inside the curve still have room to improve.

continuous batchingSlotting requests in at the token level so nothing waits for a batch to start. Like seating diners as soon as a table frees up.

Tensor Parallelism (TP)Splitting one layer's computation across several GPUs. It needs a lot of communication, so fast NVLink matters.

speculative decodingGuessing upcoming tokens and having the model verify them in one pass to generate faster. Works especially well on predictable code output.

P/D disaggregationRunning prefill (reading input) and decode (generating tokens) on separate workers. Like splitting prep and cooking in a kitchen.

NVFP4A 4-bit microscaling floating-point format that can greatly raise serving efficiency with little-to-no quality loss.