Inside vLLM: Anatomy of a High-Throughput LLM Inference System (2025)

TL;DR Highlight

A walkthrough of vLLM's scheduler, KV cache, and prefix caching internals for high-throughput serving.

Who Should Read

A backend developer serving LLMs with vLLM who wants to understand the principles behind throughput and latency tuning.

Core Mechanics

The engine core centers on the scheduler and KV cache manager
- The vLLM engine core consists of a Model Executor, a Structured Output Manager, and a Scheduler.
- The scheduler has an FCFS or priority policy plus waiting and running queues.
- The KV cache manager is the heart of paged attention.
Every step has three stages
- Schedule picks which requests run this step (decode and/or chunked prefill).
- Forward pass runs the model and samples tokens.
- Postprocess appends tokens, checks stop conditions, and returns KV blocks of finished requests.
V1 mixes prefill and decode in one step
- The V0 engine could process only prefill or only decode per step.
- The scheduler handles decode requests in the running queue first, then prefill from the waiting queue.
- Prefill is usually compute-bound while decode is memory-bandwidth-bound.
Chunked prefill stops long prompts from monopolizing steps
- Long prompt prefill is split into smaller chunks across several steps.

Community Reactions

Reading nano-vllm helps understanding, one commenter suggests
- It is said to be about 5kloc and supports just one model.
- It reportedly contains all the major pieces that make an inference engine fast.
A question about how this compares with Radix Attention
- It came with praise that the post goes beyond paged attention.
A view that other pieces mattered more than paged attention
- The commenter lists separating the web server and GPU process, continuous batching, KV caching/chunking, and a large model library with low precision.

How to Apply

Put the fixed prompt part first so prefix caching hits
- Today, move the system prompt and shared context to the very start of your vLLM prompts and append per-request content after.
- Reuse requires matching 16-token blocks, so don't put per-request values before the shared part.
Turn on chunked prefill if you serve long prompts
- If long requests cause latency spikes for others, set long_prefill_token_threshold to a positive value and compare measurements.
Use cache salt in multi-tenant setups
- Add a cache salt to requests when users must not share KV blocks.
- The salt goes into the first block hash, so only requests with the same salt reuse blocks.
Use --enforce-eager only for debugging
- Enable it to rule out CUDA graph capture effects, and keep it off when measuring latency.

Try It

copy & run

# Paste into Claude or ChatGPT chat (a checkup prompt applying the article's prefix caching idea to your service)

I serve an LLM with vLLM. vLLM's prefix caching (on by default) reuses KV cache for identical prompt prefixes by hashing 16-token blocks. It speeds up prefill only, not decode.

Below is our service's prompt template.
{{프롬프트 템플릿}}

Separate the parts that change per request from the parts that are always the same. Give me a reordered template with the constant parts first, and explain which changes would break cache reuse.

Next: send the same request twice with the new template and compare time to first token.

Terminology

Paged attentionManages the KV cache in fixed-size blocks, like OS memory pages. The blocks act as an index mapping tokens to their computed KV.

Continuous batchingInstead of running a fixed batch to the end, new and old requests are reconsidered every step. Like a restaurant seating the next guest as soon as a table frees up.

Prefill / DecodePrefill processes the whole prompt at once; decode processes only the latest token to produce the next one.

KV cacheStored key/value vectors of already-computed tokens, so they aren't recomputed from scratch each time.

CUDA graphRecords a sequence of GPU work and replays it, reducing the overhead of launching kernels one by one.